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FOREWORD 


The  Port  Knox  Field  Unit  has  been  highly  successful  in  applying  psycho¬ 
logical  principles  and  educational  technology  in  solving  Army  training 
problems.  The  training  evaluation/ feedback  team  of  this  unit  performs 
research  and  develop  ment  aimed  at  providing  the  Army  with  practical  methods 
for  assessing  and  inproving  the  effectiveness  of  its  training  programs  and 
materials. 

In  the  past  the  Army  has  not  had  an  effective  method  for  evaluating  and 
inproving  their  training  programs.  Previous  attempts  to  evaluate  training 
have  either  relied  heavily  on  the  perceptions  of  the  trainers  and  the 
soldiers  being  trained,  or  have  depended  on  the  use  of  hands-on  testing 
as  the  sole  source  of  evaluation  data.  Neither  of  these  methods  has  been 
entirely  satisfactory.  Perceptions  of  the  training  by  trainers  or  trainees 
seldom  provide  the  kinds  of  information  necessary  to  identify  and  correct 
specific  training  problems.  And  while  hands-on  testing  may  identify 
the  tasks  for  which  training  is  deficient,  hands-on  tests  usually  do  not 
provide  any  information  about  the  probable  cause  of  poor  task  performance. 
Without  information  about  probable  causes,  specific  action  cannot  be  taken 
to  eliminate  the  training  deficiencies. 

Over  a  three  year  period,  the  training  evaluation/ feedback  team  has  devel¬ 
oped  a  unique  system  for  evaluating  training  programs  that  enables  the 
evaluator  to  identify  and  correct  the  most  probable  causes  of  poor  task 
performance.  Detailed  guidance  in  using  this  new  system  is  provided 
in  a  set  of  easy-to-use  job  aids  designed  specifically  for  evaluating 
Army  training  programs.  The  Training  Program  Evaluation  (TPE)  methodology 
described  in  these  job  aids  was  developed  by  progressively  refining  proto¬ 
type  data  collection  procedures  and  formats  through  a  series  of  field  trials 
in  which  TPE  was  tested  with  typical  users  against  typical  Army  training. 
This  report  traces  the  development  and  field  trial  of  the  TPE  system  from 
its  inception  to  the  completed  TPE  job  aids.  The  information  provided  in 
this  report  on  the  field  trial  of  the  TPE  system  and  its  associated  job  aids 
may  prove  valuable  to  anyone  tasked  with  conducting  a  large  scale  training 
program  evaluation,  particularly  against  a  new  system  during  the  operational 
testing  phases  of  the  Life  Cycle  Systems  Management  Model. 


PREFACE 


This  report  traces  the  development  and  field  trial  of  a  system  for  evalu¬ 
ating  training  programs.  Although  the  report  includes  considerable  information 
on  the  evaluation  system  itself,  no  attempt  has  been  made  to  fully  describe 
the  system  or  to  demonstrate  how  it  is  used.  That  information  is  amply  docu¬ 
mented  elsewhere  (Witmer,  Note  5;  Kristiansen,  Note  6;  Kristiansen  and  Witmer, 
Note  7;  and  Kristiansen  and  Witmer,  Note  8).  Similarly,  though  much  is  said 
about  the  Ml  OT-III  in  connection  with  the  field  trial  of  the  training  evalua¬ 
tion  system,  this  report  makes  no  attempt  to  provide  a  complete  account  of  all 
the  events  associated  with  the  Ml  OT-III.  Background  information  on  the  Ml 
OT-III  is  presented  only  insofar  as  it  relates  to  the  development  and  field 
trial  of  the  Training  Program  Evaluation  (TPE)  job  aids  and  procedures. 

The  purpose  of  this  report  is  twofold:  (1)  To  trace  the  development  of 
the  TPE  system  from  its  inception,  through  the  Ml  OT-III  field  trial,  to  the 
developed  system  as  described  in  the  revised  TPE  job  aids,  and  (2)  To  explain 
how  ARl's  experiences  in  using  the  TPE  system,  particularly  during  the  Ml 
OT-III,  led  to  changes  in  the  TPE  system. 


REQUIREMENT 


In  April  1978,  the  Army  Training  Study  Group  (ARTS)  asked  ARI  to  develop 
job  aids  and  methodological  guides  for  assessing  the  effectiveness  and  effi¬ 
ciency  of  Army  training  programs-  ARTS  further  requested  that  the  Ml  tank 
system  be  used  as  a  test  bed  to  test  the  adequacy  of  any  materials  or  method¬ 
ologies  developed.  This  report  documents  the  process  by  which  ARI  developed 
and  field  tested  the  requested  job  aids  and  methodologies. 

PROCEDURE 

A  system  for  assessing  the  effectiveness  and  efficiency  of  training  pro¬ 
grams,  henceforth  referred  to  as  Training  Program  Evaluation  (TPE) ,  was  de¬ 
veloped  by  alternately  designing,  trying  out,  and  revising  data  collection 
formats  and  procedures.  ARI's  initial  response  to  the  ARTS  request,  which  was 
to  develop  a  Training  Observation  Form  and  a  Training  Opinion  Questionnaire 
for  evaluating  training  during  the  Ml  OT-II  indicated  the  need  for  additional 
development  of  training  evaluation  materials.  TPE  used  as  its  starting  point 
the  procedures  and  data  collection  formats  developed  for  ARI  by  Harless  Per¬ 
formance  Guild,  Inc.  These  procedures  and  formats,  known  as  the  Harless  Guide¬ 
lines,  were  tested  against  several  courses  routinely  conducted  at  the  Armor 
Center  with  encouraging  results.  The  early  trials  of  the  Harless  Guidelines 
at  the  Armor  Center,  while  encouraging,  indicated  that  additional  work  was  re¬ 
quired  to  develop  TPE  to  the  point  that  it  could  be  used  routinel.  by  Army 
personnel  to  effectively  evaluate  training. 

After  further  modifications  by  ARI,  the  TPE  system  was  ready  to  undergo 
the  major  test  of  its  usefulness  as  a  method  for  evaluating  training  programs. 
TPE  was  to  be  used  to  evaluate  the  effectiveness  and  efficiency  of  the  training 
designed  to  transition  soldiers  from  the  M60A1  tank  to  the  Ml  tank  during  the 
Ml  OT-III.  An  ARI  researcher  was  present  on  site  during  the  OT  to  analyze  the 
TPE  data  and  assess  the  usefulness  of  TPE  in  evaluating  the  Ml  transition 
training  program.  Based  on  lessons  learned  from  the  OT-III  experience,  ARI 
further  revised  the  TPE  materials. 

FINDINGS 

The  information  provided  by  the  TPE  system  during  the  Ml  OT-III  was 
found  useful  by  all  of  the  agencies  involved  in  the  evaluation  of  the  Ml  transi 
tion  training  program.  The  data  provided  by  the  TPE  system  were  used  to  recom¬ 
mend  changes  in  the  transition  training  program.  Many  of  the  recommended 
changes  were  adopted  by  the  trainers,  with  resulting  increases  in  soldier  pro¬ 
ficiency  on  the  end-of-block  tests.  Some  of  the  NCO's  collecting  the  TPE  data 
during  the  field  trial  failed  to  make  the  necessary  observations  required  by 
the  TPE  system,  opting  instead  to  record  their  subjective  impressions  of  the 
training.  To  remedy  this  situation,  the  TPE  materials  were  revised  following 
the  field  trial  in  a  way  that  encourage,  data  collectors  to  record  only  what 
they  observe  rather  than  record  their  opinions  about  the  training.  The  revised 
TPE  system  is  documented  in  a  set  of  four  user-oriented  job  aids  (ARI  Research 
Products  81-15,  81-16,  81-17,  and  81-18). 


UTILIZATION 


The  information  provided  in  this  report  may  be  useful  to  military  managers 
who  are  given  the  responsibility  to  evaluate  existing  training  programs.  The 
report  traces  the  development  and  field  trial  of  the  TPE  system  leading  to  the 
finished  product  —  the  revised  TPE  job  aids.  It  points  out  the  advantages  of 
TPE  as  an  evaluation  method,  and  identifies  common  pitfalls  to  avoid  when  ap¬ 
plying  an  evaluation  system  such  as  TPE.  The  lessons  learned  during  the  field 
trial  of  the  TPE  system  may  help  military  managers  avoid  costly  mistakes  in 
conducting  their  own  training  program  evaluations. 


TABLE  OF  CONTENTS 

Section  Page 

I.  GENESIS  OF  THE  TPE  SYSTEM  I 

Introduction  1 

The  Ml  OT-II  Training  Evaluation  2 

Contracting  for  the  Development  of  TPE  Materials  2 

II.  DEVELOPMENT  AND  TRIAL  OF  THE  ORIGINAL  TPE  JOB  AIDS  4 

The  Harless  Guidelines  4 

Using  the  Harless  Guidelines  4 

User  Acceptance  of  the  Harless  Guidelines  5 

Lessons  Learned  Using  the  Harless  Guidelines  5 

A  Job  Aid  for  Correcting  Training  Program  Deficiencies  6 

III.  PLANNING  THE  Ml  OT-III  TRAINING  EVALUATION  7 

Defining  the  Scope  and  Range  of  Evaluation  Activities  7 

Outlining  Data  Requirements  for  the  Evaluation  8 

Planning  the  Data  Collection  9 

Management  of  the  Data  Flow  9 

Diagnostic  Tests  10 

Scheduling  10 

Further  Plans  11 

Pilot  Test  of  the  OT-III  Data  Collection  Forms  11 

TCATA's  Detailed  Test  Plan  11 

IV.  EVALUATING  THE  Ml  TRANSITION  TRAINING  LESSON  PLANS  13 

Obtaining  the  Lesson  Plans  13 

Review  and  Initial  Evaluation  of  Lesson  Plans  13 

A  Systematic  Approach  to  Lesson  Plan  Evaluation  13 

Revised  Lesson  Plans  14 


ix 


Section 


V.  DESIGNING  THE  TPE  WORKSHOPS  AND  JOB  AIDS 
FOR  THE  Ml  OT-III  EVALUATION 

VI.  FIELD  TRIAL  OF  THE  TPE  SYSTEM 

Scope  and  Purpose  of  the  Field  Trial 

Conducting  the  Field  Trial 

Field  Trial  Assessment  of  the  TPE  System 

VII.  REVISING  THE  TPE  SYSTEM  ON  THE  BASIS  OF 

PROBLEMS  IDENTIFIED  DURING  THE  FIELD  TRIAL 

Motivation  of  the  Data  Collectors 

Data  Flow  Problems  and  Suggested  Solutions 

Problems  in  Using  the  Data  Collection  Forms 

Revision  of  the  Data  Collection  Forms 

The  Revised  Job  Aids 

VIII.  CONCLUDING  REMARKS 

IX.  REFERENCE  NOTES 

X.  REFERENCES 

APPENDIX  A  Training  Observation  Form 

APPENDIX  B  Training  Opinion  Questionnaire 

APPENDIX  C  TPE  Data  Collection  Forms  Used  During  the 

Evaluation  of  the  Ml  OT-III  Transition  Training 

APPENDIX  D  Other  Forms  Used  During  the  Ml  OT-III  for 

Collecting  Training  Effectiveness  Information 

APPENDIX  E  Lesson  Plan  Evaluation  Criteria 


SECTION  I 


Introduction 


GENESIS  OF  THE  TPE  SYSTEM 


The  Army  recognizes  the  need  to  evaluate  its  training  programs.  Due, 
however,  to  the  unavailability  of  personnel,  resources,  and  expertise  in  evalu¬ 
ation  methodology,  formal  evaluation  programs  for  use  in  training  course  im¬ 
provement  are  not  a  standard  feature  of  the  Army  training  system.  In  1976, 
a  survey  of  training  program  evaluation  activities  in  the  military  services 
indicated  that,  in  the  Army,  individual  schools  did  little  to  assess  the  effec¬ 
tiveness  of  the  courses  they  conducted  (Hall,  Lam  &  Bellamy,  1976).  In  1981 
the  Comptroller  General  of  the  United  States,  in  a  report  to  the  Congress,  said 
"Because  the  Army  does  not  have  an  effective  Army-wide  management  system  to 
oversee  the  skill  training  program,  it  is  difficult  to  identify  where  improve¬ 
ments  are  needed.  An  effective  monitoring  and  evaluation  system  is  needed  to 
provide  Army  commanders  at  all  levels  the  program  evaluation  data  and  other 
management  information  needed  for  informed  decision  making."  (GAO  Report 
FPCD-81-29,  1981.) 

The  training  program  evaluation  (TPE)  system  discussed  in  this  report  can 
be  used  by  Army  personnel  to  evaluate  the  effectiveness  and  efficiency  of  Army 
training  programs.  The  TPE  system  identifies  specific  training  program  de¬ 
ficiencies,  allowing  the  program  evaluator  to  identify  the  most  likely  causes 
of  poor  soldier  performance.  The  TPE  system  also  assists  training  managers  in 
correcting  training  program  problems  by  recommending  courses  of  actions  to  be 
taken  by  the  training  manager  in  the  event  that  particular  deficiencies  are 
discovered.  The  TPE  system  is  unique  in  that  it  does  not  rely  on  the  percep¬ 
tions  of  the  soldiers  being  trained  or  trainers  for  its  data.  Neither  does  it 
require  the  administration  of  special  post-tests  to  assess  the  effectiveness 
of  training.  TPE  examines  the  training  program  directly,  comparing  the  train¬ 
ing  program  as  it  is  documented  in  the  lesson  plan  and  delivered  by  the  in¬ 
structor  to  characteristics  that  according  to  educational  technologists  are 
indicative  of  a  good  training  program.  The  TPE  system  requires  that  an  ob¬ 
server  be  present  on  site  during  the  delivery  of  training  and  during  the  con¬ 
duct  of  the  end-of-block  tests  to  collect  information  about  the  instructional 
procedures  used  in  training  and  testing  the  soldiers.  The  information  obtained 
during  this  observation  phase  is  used  in  conjunction  with  information  from  the 
lesson  plans  and  soldier  performance  data  as  measured  by  end-of-block  tests  to 
identify  training  program  deficiencies  and  their  probable  causes.  By  identify¬ 
ing  causes  of  training  program  deficiencies,  TPE  allows  the  Army  to  eliminate 
program  deficiencies  by  removing  the  causes  of  those  deficiencies.  TPE  ful¬ 
fills  the  Army's  need  for  an  effective  means  of  monitoring  and  evaluating  train 
ing  programs.  Complete  information  about  the  TPE  system  and  how  it  is  used  is 
documented  in  a  series  of  job  aids  (Witmer,  Note  5;  Kristiansen,  Note  6; 
Kristiansen  and  Witmer,  Note  7;  and  Kristiansen  and  Witmer,  Note  8). 

The  evaluation  methodology  incorporated  in  the  TPE  system  was  developed 
in  response  to  the  Army's  need  to  evaluate  the  effectiveness  and  efficiency  of 
training  programs  for  new  weapons  systems.  Army  planners  realized  that  a  large 


number  of  new  weapons  systems  were  under  development  and  would  be  fielded  in 
the  near  future.  Planners  such  as  the  Army  Training  Study  Group  (ARTS)  foresaw 
that  the  introduction  of  the  new  weapons  systems  would  involve  large-scale 
training  programs  designed  to  train  soldiers  to  operate  and  maintain  the  weapons 
systems.  ARTS  knew  that  the  overall  effectiveness  of  the  weapons  systems  was 
linked  inalterably  to  the  effectiveness  of  the  training  programs  developed  for 
them.  This  was  a  source  of  concern  for  ARTS  because  adequate  guidelines  for 
evaluating  Army  training  programs  did  not  exist  at  the  time. 

The  Ml  OT-II  Training  Evaluation 

In  April  1978,  the  Army  Research  Institute  (ARI)  received  a  mid-year  re¬ 
quest  from  ARTS  to  develop  job  aids  and  methodological  guides  for  assessing 
the  effectiveness  and  efficiency  of  Army  training  programs.  ARTS  further  re¬ 
quested  that  the  Ml  Tank  system  be  used  as  a  test  bed.  The  Ml  was  a  major 
weapon  system  that  was  already  well  into  the  Operational  Testing  (OT)  Cycle. 

The  individual  training  phase  of  the  Ml  OT-II  was  about  to  begin.  In  order  to 
take  advantage  of  the  opportunity  presented  by  the  Ml  OT-II,  ARI  quickly  as¬ 
sembled  two  data  collection  instruments  to  be  used  in  evaluating  the  individual 
in-turret  portion  of  the  training  package  for  transitioning  soldiers  from  the 
M60A1  tank  to  the  Ml  tank.  A  Training  Observation  Form  (Appendix  A)  was  pre¬ 
pared  to  structure  the  observation  of  training.  In  response  to  a  request  from 
the  US  Army  Armor  Center  (USAARMC) ,  a  Training  Opinion  Questionnaire  (Appendix 
B)  was  developed  to  assess  the  opinions  and  attitudes  of  the  OT-II  player  per¬ 
sonnel  regarding  the  training  during  different  phases  of  the  OT. 

The  Training  Observation  Form  and  the  Training  Opinion  Questionnaire  were 
piloted  against  the  Ml  OT-II  transition  training  on  site  at  Fort  Bliss,  Texas, 
by  a  team  of  ARI  researchers.  Following  the  OT,  the  data  collected  using 
these  evaluation  instruments  were  analyzed  in  an  attempt  to  determine  which  of 
the  data  collected  were  useful  and  which  were  not.  The  findings  regarding  the 
usefulness  of  the  evaluation  data  were  shared  with  USAARMC  and  ARTS. 

The  data  collected  during  the  OT-II  was  not  adequate  for  evaluation  pur¬ 
poses,  due  in  part  to  the  unavailability  of  individual  soldier  performance  data. 
Few  conclusions  could  be  drawn  on  the  basis  of  the  training  observation  and 
training  opinion  data  alone.  Detailed  analysis  of  the  training  opinion  data 
indicated  that  it  was  virtually  useless  as  a  measure  of  training  effectiveness. 
Because  of  the  limited  usefulness  of  the  evaluation  instruments  developed  for 
the  Ml  OT-II,  ARI  and  ARTS  decided  that  an  extensive  developmental  effort  aimed 
at  producing  a  systematic  method  for  conducting  training  program  evaluations 
was  needed. 

Contracting  for  the  Development  of  TPE  Materials 

Toward  the  end  of  FY  78,  ARTS  requested  that  ARI  write  a  statement  of  work 
for  a  contract  to  develop  TPE  materials.  The  contract,  to  be  let  sole  source 
to  Harless  Performance  Guild,  Inc.,  was  to  be  funded  by  monies  secured  by  ARTS 
from  TRADOC,  monitored  jointly  by  the  ARI  Field  Unit  at  Fort  Knox  and  the 
USAARMC  Directorate  of  Training  Developments  (DTD)  and  let  by  the  Fort  Knox 
Procurement  Division  of  the  Directorate  of  Industrial  Operations.  The  state¬ 
ment  of  work  was  written  and  the  contract  to  develop  TPE  materials  was  let. 


Under  the  terms  of  the  contract,  Harless  was  required  to  develop  guide¬ 
lines  and  job  aids  for  evaluating  the  effectiveness  of  training  designed  for 
the  operational  testing  of  the  Ml  tank.  Using  lessons  learned  during  the  OT-II 
the  guides  and  job  aids  were  to  be  designed  so  that  they  could  be  used  by  ARI 
researchers  to  evaluate  training  during  the  Ml  OT-III.  The  guides  and  job  aids 
were  to  include  their  own  training  materials  so  that  formal  training  in  their 
use  would  not  be  necessary.  Although  specifically  designed  for  ARI  researchers 
to  use  in  evaluating  Ml  transition  training,  it  was  understood  that  the  method¬ 
ology  developed  should  be  generalizable  to  any  training  program  associated  with 
the  introduction  of  a  new  weapons  system  into  the  Army. 


SECTION  II 


DEVELOPMENT  AND  TRIAL  OF  THE  ORIGINAL  TPE  JOB  AIDS 


The  Harless  Guidelines 


The  primary  product  to  come  out  of  the  contract  with  Harless  Performance 
Guild,  Inc.  was  ARI  Working  Paper,  FKFU  80-1,  "Guidelines  for  Conducting  a 
Training  Program  Evaluation."  This  document,  completed  in  November  1979 
went  a  long  way  towards  fulfilling  the  Army's  need  for  a  systematic  method 
for  evaluating  training  programs  for  new  weapon  systems.  The  "Guidelines  for 
Conducting  a  Training  Program  Evaluation"  (Harless,  Note  1)  henceforth  to  be 
referred  to  as  the  Harless  guidelines,  divided  training  program  evaluation 
into  five  phases.  In  Phase  1,  plans  are  made  for  conducting  the  TPE.  During 
this  phase,  all  the  background  information,  training  materials,  evaluation 
instruments,  and  task  documentation  needed  to  perform  the  TPE  are  assembled 
and  a  plan  for  the  conduct  of  the  TPE  is  prepared.  Data  for  determining  the 
effectiveness  of  the  training  program  is  collected  during  Phase  2.  Data  are 
collected  to  describe  the  actual  training  process,  testing  process,  training 
environment,  trainee  characteristics,  instructor  characteristics,  and  instruc 
tor  and  trainee  reactions  to  the  training.  Phase  3  consists  of  summarizing 
and  analyzing  the  in-course  test  data  collected  in  Phase  2.  An  analysis  of 
the  test  data  collected  in  Phase  3  identifies  the  tasks  on  which  test  per¬ 
formance  is  deficient  and  those  tasks  which  require  further  investigation. 

In  Phase  4,  training  modules  which  warrant  further  investigation  are  examined 
to  determine  if  the  evidence  collected  during  the  previous  phases  suggests 
that  the  performance  deficiencies  are  due  to  training  or  derive  from  other 
causes.  The  findings  of  the  TPE  are  documented  in  Phase  5.  In  this  phase,  a 
report  outlining  the  steps  followed  during  the  TPE  and  the  conclusions 
reached  is  written  for  the  sponsor  of  the  evaluation.  For  Phases  1-4,  the 
Harless  guidelines  provide  detailed  worksheets  for  recording  the  required  in¬ 
formation.  Step-by-step  directions  for  completing  the  worksheets  are  also 
provided. 

Using  the  Harless  Guidelines 

ARI,  Fort  Knox  had  worked  closely  with  Harless  during  the  development  of 
the  Harless  guidelines  and  was  aware  of  the  potential  utility  of  the  TPE 
approach  described  in  the  guidelines.  ARI  researchers  initially  tested  the 
usefulness  of  some  of  the  observation  worksheets  on  institutional  training 
that  is  routinely  conducted  at  the  Armor  Center.  The  guidelines  were  piloted 
against  Armor  Officer  Basic  (AOB)  courses  in  supply  management  and  forward 
observer  procedures,  and  courses  in  platoon  sergeants'  motivation,  M60A1 
track  removal/installment,  and  mechanical  training  for  two  different  machine- 
guns.  From  these  early  trials  of  the  Harless  guidelines,  it  was  clear  that 
the  worksheets  used  for  the  observation  phase  were  a  valuable  aid  in  identify 
ing  training  and  testing  problems.  It  was  also  clear  that  the  methodology 
described  in  the  guidelines  could  be  applied  to  almost  any  performance-based 
training  and  need  not  be  confined  to  evaluating  transition  training  for  new 
weapon  systems. 


From  ARI's  early  experiences  with  the  Harless  guidelines,  it  became  appar¬ 
ent  that  a  certain  amount  of  skill  is  required  to  complete  the  various  work¬ 
sheets  in  sufficient  detail,  and  that  this  skill  improves  with  practice.  After 
using  the  worksheets  in  several  different  Armor  Center  courses,  ARI  researchers 
found  that  they  became  more  proficient  in  making  the  necessary  observations  and 
in  recording  their  observations  on  the  worksheets.  To  ensure  that  persons  and 
organizations  wishing  to  use  the  Harless  guidelines  received  familiarization 
with  and  practice  in  using  the  worksheets  prior  to  conducting  any  formal  evalu¬ 
ations,  ARI  instituted  the  procedure  of  conducting  a  short  workshop  on  TPE 
methodology  for  potential  users. 

In  the  workshop,  the  purpose  of  TPE  and  the  approach  described  in  the 
guidelines  is  explained.  The  user  is  then  stepped  through  the  worksheets, 
item-by-item.  When  the  user  is  thoroughly  familiar  with  the  worksheets  and 
method,  he/she  then  practices  using  the  worksheets  in  evaluating  an  operational 
three-  or  four-hour  block  of  instruction.  Upon  completion  of  the  practice 
exercise,  the  user  shares  observations  with  those  of  other  workshop  partici- 
tants  and  receives  feedback  from  the  workshop  leader. 


User  Acceptance  of  the  Harless  Guidelines 


The  Harless  guidelines  were  well  received  by  the  Army  and  the  need  for 
the  methodology  described  in  the  guidelines  was  reaffirmed  by  the  many  requests 
that  ARI  received  for  copies  of  the  guidelines  and  to  conduct  workshops  in 
their  use.  Workshops  have  been  conducted  for  the  USAARMC  Staff  and  Faculty 
Training  Division  and  Directorate  of  Training  Developments  (DTD),  the  Office 
of  Armor  Force  Management  and  Standardization  (OAFMS) ,  and  the  US  Army  Armor 
and  Engineer  Board.  Other  organizations  receiving  workshops  prior  to  the  Ml 
OT-III  were  the  Directorate  of  Evaluation  at  Fort  Benjamin  Harrison,  TRADOC 
Combined  Arms  Test  Activity  (TCATA)  at  Fort  Hood,  and  ARI  researchers  and 
civilian  contractors  at  Fort  Bliss  in  connection  with  the  proposed  evaluation 
of  new  equipment  training  (NET)  for  developing  Air  Defense  Weapon  Systems.  Of 
these,  the  most  frequent  user  of  the  Harless  guidelines  has  been  OAFMS.  OAFMS 
has  the  mission  of  evaluating  the  state  of  training  in  the  Armor  force  and  has 
used  the  guidelines  to  help  satisfy  its  role  as  the  Directorate  of  Evaluation 
at  Fort  Knox.  In  addition,  the  USAARMC  DTD  used  the  Harless  guidelines  to 
certify  the  NET  program  and  prepare  Operational  Test  Readiness  Statements 
(OTRS)  on  the  Cavalry  Fighting  Vehicle  (CFV)  OT-II  NET  player  personnel. 


Lessons  Learned  in  Using  the  Harless  Guidelines 


Several  lessons  were  learned  from  conducting  the  workshops  for  the  various 
organizations  and  in  providing  assistance  to  OAFMS  during  the  evaluation  of  the 
CFV  and  Advanced  NCO  Course  training.  The  first  lesson  learned  was  that  some 
individuals  are  much  more  proficient  in  using  the  Harless  Guidelines  to  objec¬ 
tively  evaluate  training  than  are  others.  Some  observers  tend  to  use  the 
Harless  worksheets  to  record  their  subjective  impressions  of  the  training  or 
trainer,  rather  than  perform  the  more  demanding  task  of  making  the  specific 
observations  called  for  by  the  Harless  Guidelines.  Other  observers,  viewing 
the  same  training,  utilize  the  worksheets  correctly,  identifying  important 
training  problems.  This  suggests  that,  whenever  possible,  prospective  data 
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collectors  should  be  screened  in  order  to  eliminate  those  who  for  one  reason 
or  another  do  not  use  the  Harless  worksheets  correctly.  The  practice  exercise 
during  the  TPE  workshop  provides  an  excellent  opportunity  to  identify  individ¬ 
uals  who  do  or  do  not  make  good  training  observers.  A  second  lesson  learned 
was  that  while  the  guidelines  and  accompanying  workshop  were  usually  effective 
in  teaching  potential  users  to  plan  the  TPE,  collect  the  data  and  identify 
possible  causes  of  performance  deficiencies,  it  was  somewhat  less  effective  in 
training  users  to  draw  conclusions  and  make  recommendations  for  training  pro¬ 
gram  changes  based  on  the  data  collected.  The  difficulties  that  potential 
users  experienced  in  drawing  conclusions  and  making  recommendations  suggested 
the  need  for  additional  guidance  in  the  analysis  and  conclusion  phases  (Phases 
3,  4,  and  5)  of  the  Harless  guidelines. 

A  Job  Aid  for  Correcting  Training  Program  Deficiencies 


ARI  was  not  surprised  that  early  users  of  the  Harless  guidelines  experi¬ 
enced  difficulties  in  drawing  conclusions  and  making  recommendations  based  on 
their  observations  of  training.  ARI  had  already  discovered  that  the  procedures 
outlined  in  Phase  4  of  the  Harless  guidelines  did  not  lead  typical  military 
users  to  a  clear  identification  of  the  causes  of  a  given  performance  deficiency 
Additionally,  the  guidance  provided  in  Phase  5  was  insufficient  for  an  un¬ 
initiated  user.  Furthermore,  Harless  had  not  been  tasked  to  include  guidance 
to  the  user  on  how  to  recommend  program  changes  based  on  performance  defi¬ 
ciencies;  the  tasking  had  only  been  to  provide  a  method  for  identifying  those 
deficiencies  and  their  possible  causes.  Procedures  for  correcting  performance 
deficiencies  was  to  be  the  subject  of  a  second  developmental  effort. 

In  the  last  quarter  of  FY  79,  a  contract  was  let  to  Seville  Research  Cor¬ 
poration  to  develop  guidelines  for  specifying  modifications  to  training  on  the 
basis  of  training  problems  discovered  during  the  evaluation  of  training.  The 
final  product,  entitled  "Methodology  for  Correcting  Deficiencies  in  Training 
Programs,"  was  completed  by  Seville  Research  Corporation  in  April  1980  (Spears, 
Maxey  and  Roush,  Note  2) . 


Like  the  Harless  guidelines  the  Seville  product  was  designed  to  be  used  by 
ARI  researchers  to  evaluate  transition  training  during  the  Ml  OT-III.  The 
Harless  guidelines  were  to  be  used  to  observe  training  and  testing  in  order  to 
determine  the  causes  of  performance  deficiencies.  The  Seville  product  would 
complete  the  evaluation  process  by  specifying  fixes  for  the  problems  identified 
by  the  Harless  guidelines.  Although  the  contract  with  Seville  called  for  a 
document  compatible  with  the  Harless  guidelines,  the  Seville  product  did  not 
interface  well  with  the  Harless  guidelines.  Further  work  was  required  to  gen¬ 
erate  a  methodology  for  correcting  deficiencies  that  would  be  compatible  with 
the  TPE  methodology  described  in  the  Harless  guidelines. 


SECTION  III 


PLANNING  THE  Ml  OT-III  TRAINING  EVALUATION 


Defining  the  Scope  and  Range  of  Evaluation  Activities 

Planning  for  the  Ml  OT-III  began  early.  The  OT-III  was  to  focus  on  re¬ 
solving  the  training  and  maintenance  issues  unanswered  by  Ml  OT-II.  Just  how 
this  was  to  be  accomplished  was  decided  in  a  series  of  meetings  among  the  par¬ 
ticipating  organizations:  the  TRADOC  Combined  Arms  Test  Activity  (TCATA) ,  the 
Operational  Test  and  Evaluation  Agency  (OTEA) ,  the  Armor  Center  Directorate  of 
Training  Developments  (DTD),  the  Office  of  Armor  Force  Management  and  Standard¬ 
ization  (OAFMS),  the  TRADOC  Systems  Analysis  Activity  (TRASANA) ,  and  ARI. 

In  the  early  meetings  of  the  above  mentioned  organizations  few  decisions 
were  reached.  The  meetings  served  primarily  as  a  forum  for  each  organization 
to  state  its  views  and  data  needs  to  the  other  participants.  For  example,  in 
the  29  March  1979  meeting,  OTEA  stated  its  preference  for  obtaining  user  per¬ 
formance  data  as  the  measure  of  training  effectiveness,  while  TCATA  argued  for 
a  more  direct  measure  of  training  effectiveness.  OTEA  suggested  that  the  OAFMS/ 
TRASANA/DTD/ARI  training  effectiveness  analysis  be  accepted  by  TCATA,  but  TCATA 
insisted  that  it  preferred  to  conduct  its  own  TEA.  Among  the  other  issues 
discussed  during  the  29  March  meeting  was  whether  the  OT-III  should  be  conducted 
at  Fort  Knox  or  Fort  Hood.^ 

In  June  1979,  OTEA  drafted  a  test  plan  outlining  the  scope  of  the  Ml 
OT-III.  The  OT-III  was  to  be  a  three-phased  test  conducted  by  TCATA  from  June 
1980  through  April  1981.  Ml  tanks  were  to  be  delivered  incrementally,  begin¬ 
ning  with  three  in  June  and  continuing  until  a  total  of  55  had  been  issued  by 
the  end  of  November  1980.  During  Phase  I  (transition  training),  a  tank  bat¬ 
talion  would  be  transitioned  company  by  company  from  the  M60  series  tank  to  the 
Ml  tank.  Transition  training,  to  be  conducted  at  Fort  Hood  by  the  Ml  New 
Equipment  Training  Team  (NETT),  would  consist  of  an  orientation  for  staff  per¬ 
sonnel,  individual  and  collective  tank  crew  skill  training,  and  organizational 
maintenance  personnel  training.  Training  for  DS/GS  maintenance  personnel  was 
to  be  given  at  the  US  Army  Ordnance  School  at  APG,  MD.  Phase  II,  consisting  of 
a  live  firing  exercise  under  simulated  combat  conditions  integrated  with  maneu¬ 
ver  exercises,  would  be  used  to  determine  the  mission  and  system  reliability 
for  the  test.  Phase  III  would  involve  up  to  a  tank  battalion  task  force  en¬ 
gaged  in  non-firing  exercises  against  an  aggressor  force  (up  to  a  brigade  size). 
Data  would  be  collected  over  a  series  of  field  training  exercises  to  include 
offensive,  defensive  and  retrograde  operations  and  would  address  the  issues  of 
logistical  supportability ,  force  effectiveness  training,  and  f ightability . 

The  scope  of  the  OT-III  was  further  defined  in  a  memorandum  from  the 
TRADOC  Systems  Manager  for  the  Ml  to  OAFMS,  DTD,  ARI,  Directorate  of  Armor  Doc¬ 
trine  and  the  Directorate  of  Combat  Developments.  With  regard  to  training  ef¬ 
fectiveness,  the  memo  stated  that  the  Ml  NET  team  would  conduct  individual 
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Fort  Hood  was  later  chosen  as  the  site  for  the  OT-III. 


training  for  MOS  19K/L  tank  crewmen,  collective  training  for  Ml  crews  through 
platoon  gunnery  qualification,  and  organizational  level  training  for  MOS  45E 
and  MOS  63E  mechanics.  Tactical  training  would  be  conducted  by  the  unit.  The 
effectiveness  of  the  proposed  programs  for  individual  and  collective  training, 
to  include  tactical  training  were  to  be  evaluated.  Effectiveness  of  the  pro¬ 
gram  of  instruction  (POI),  training  aids,  training  devices  and  training  liter¬ 
ature  would  be  assessed.  Criterion-referenced  performance  testing  conducted 
at  the  end  of  each  block  of  instruction  during  the  training  and  at  selected 
points  during  the  test  were  to  be  used  to  determine  the  effectiveness  of  the 
training  program. 

ARI  wrote  a  memorandum  in  response  to  the  memorandum  from  the  Ml  TRADOC 
Systems  Manager.  ARI  pointed  out  that  the  data  collected  on  the  adequacy  of 
the  POI,  training  aids,  training  literature,  etc.  must  be  detailed  to  be  use¬ 
ful,  but  that  TCATA  had  not  yet  allocated  the  resources  to  collect  detailed 
training  data.  ARI  stressed  that  end-of-block  performance  data  alone  is  not 
an  adequate  measure  of  performance  since  only  a  small  subset  of  tasks  trained 
are  actually  tested  on  the  end-of-block  tests.  It  was  suggested  that  the  re¬ 
sources  to  conduct  a  detailed  evaluation  be  put  at  the  disposal  of  DTD. 

Between  6  and  10  August  1979,  a  series  of  meetings  was  held  between  TCATA, 
DTD,  the  Ml  NET  team  and  ARI.  The  primary  topic  of  discussion  was  "what  data 
should  be  collected  for  the  test  battalion,  who  would  collect  the  data,  and 
who  would  be  allowed  access  to  the  data.  DTD  had  been  tasked  by  the  CG, 

USAARMC  to  develop  diagnostic  tests  for  determining  the  readiness  posture  of 
the  test  battalion  on  the  M60A1  tank  prior  to  the  beginning  of  transition 
training  on  the  Ml  tank.  It  was  decided  that  USAARMC  would  administer  the 
diagnostic  tests  and  score  them.  TCATA  agreed  to  collect  demographic  data  on 
the  players  and  to  collect  hit/raiss  and  engagement  time  data  for  the  firing 
exercises.  The  issue  of  what  agencies  should  be  allowed  access  to  the  data 
generated  during  the  OT-III  was  raised  but  was  not  resolved  during  the  meeting. 

Outlining  Data  Requirements  for  the  Evaluation 

Subsequent  OT-III  planning  meetings  in  which  ARI  took  part  were  concerned 
solely  with  training  evaluation.  The  first  such  meetings  took  place  during  the 
period  27-30  November  1979  and  involved  representatives  of  TCATA,  ARI,  TRASANA, 
and  DTD.  The  primary  topics  of  discussion  were  the  role  of  each  agency  in  the 
evaluation,  data  collection  requirements  and  methods,  and  data  reduction.  It 
was  agreed  that  TCATA  would  coordinate,  monitor  and  control  the  data  collection 
effort  for  Ml  OT-III.  Each  of  the  participating  organizations  outlined  their 
data  collection  requirements  for  the  test.  ARI  surfaced  the  need  to  observe 
training  and  testing  directly  in  order  to  collect  observations  about  the  train¬ 
ing  and  testing  process,  and  the  requirement  to  obtain  individual  soldier  per¬ 
formance  data  on  the  tests  administered  after  each  block  of  instruction.  ARI's 
data  collection  requirements  (which  were  limited  to  data  on  the  TPE  process; 
not  on  the  evaluation  of  the  Ml  tank  system)  were  included  in  USAARMC's  data 
collection  plan,  which  was  drafted  unchanged  into  the  TCATA  Training  and  Human 
Factors  Data  Requirements.  TRASANA  stated  that  it  would  need  the  following 
pre-  and  post-diagnostic  tests  in  order  to  justify  its  participation  in  the 


OT-III  evaluation:  1)  an  SQT  type  test  of  all  M60A1  crewmen  and  mechanics  who 


would  be  undergoing  transition  training,  2)  an  SQT  type  test  given  to  all 
mechanics  and  crewmen  on  the  Ml  at  the  completion  of  transition  training,  and 
3)  an  SQT  type  test  as  described  in  2)  above  but  given  at  the  end  of  Phase  III. 

Planning  the  Data  Collection 

During  the  November  meetings,  the  group  reviewed  the  roster  of  scheduled 
events  for  Phases  I,  II,  and  III  to  determine  when  and  how  the  data  could  be 
collected.  It  was  decided  that  every  training  event  should  be  monitored  by  a 
data  collector.  For  this  to  occur,  the  presence  of  one  data  collector  per  tank 
would  be  required  during  each  training  event.  Though  the  bulk  of  the  training 
data  would  be  collected  by  these  tank  data  collectors.  Test  Team  Evaluation/ 
Supervision  personnel  from  OAFMS,  DTD,  or  ARI  would  supervise  the  data  collec¬ 
tion,  collect  data  as  needed,  and  analyze  the  data  provided  by  the  tank  data 
collectors. 

Having  made  some  determination  about  what  data  to  collect  and  when  and  how 
it  might  be  collected,  the  various  agencies  divided  up  the  responsibility  for 
the  design  and  development  of  the  data  collection  forms.  ARI  agreed  to  work 
with  USAARMC  in  preparing  data  collection  forms  for  the  observation  of  training 
and  testing.  TRASANA  agreed  to  provide  demographic  data  collection  forms,  and 
DTD,  USAARMC  accepted  responsibility  for  designing  interview  and  questionnaire 
data  collection  instruments.  TCATA  agreed  to  provide  some  special  forms  for 
recording  collective  training  data.  It  was  agreed  that  the  data  collection 
instruments  developed  by  the  various  agencies  would  be  debugged  during  the 
five-tank  Low  Rate  Initial  Production  RAM  test  to  be  conducted  at  Fort  Knox 
prior  to  the  beginning  of  the  OT-III. 

The  next  series  of  meetings  occurred  during  the  period  of  7-11  January 
1980.  The  participants  were  the  same  as  in  the  previous  meeting,  except  that 
a  representative  from  OAFMS  was  also  present.  The  issues  covered  reflected  the 
topics  of  the  previous  meeting.  The  meeting  began  with  a  review  of  each  agency's 
data  requirements  to  ensure  that  each  data  element  was  required  by  at  least  one 
of  the  participating  agencies.  From  this  review  a  final  list  was  prepared  con¬ 
sisting  of  the  combined  data  requirements  of  the  agencies  participating  in  the 
evaluation.  Work  continued  on  the  development  of  specific  data  collection  forms 
to  be  used  during  the  OT-III  training  evaluation.  Each  agency  had  brought  the 
forms  to  the  meeting  that  they  thought  they  would  need  to  collect  the  required 
data.  Under  the  guidance  of  ARI-Fort  Knox,  the  number  of  forms  being  proposed 
was  reduced  considerably.  All  agencies  agreed  to  use  the  ARI-Fort  Knox  forms 
(closely  resembling  Worksheets  A2,  A4,  B4,  B5,  and  B6  in  the  Harless  guidelines) 
for  the  observation  of  training.  Having  reached  some  agreement  on  the  forms 
that  would  be  used  to  collect  the  data,  the  five  agencies  involved  began  out¬ 
lining  specific  plans  for  the  organization  and  control  of  the  OT-III  data  col¬ 
lection  effort.  The  duties  and  responsibilities  of  each  agency  were  outlined. 
ARl's  responsibility  was  limited  to  collection  and  analysis  of  training  effec¬ 
tiveness  data  for  the  MOS  19  K/L  tank  crewmen  training. 

Management  of  the  Data  Flow 

In  March  1980,  ARI  went  to  Fort  Hood  to  conduct  a  two-day  workshop  in  ARI 
TPE  methodology  for  TCATA's  Ml  OT-III  primary  data  collection  managers.  In  a 
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meeting  following  the  workshop,  ARI  and  TCATA  devised  a  data  flow  network,  file, 
and  retrieval  system  to  be  used  for  data  handling  during  the  OT-III.  It  was 
decided  that  the  data  collection  forms  containing  the  raw  data  would  be  repro¬ 
duced  and  distributed  on  the  basis  of  need-to-know  to  the  appropriate  persons 
and  agencies.  Some  of  the  forms  would  go  to  the  data  management  team  for  entry 
into  the  computer  data  base.  Others  would  circulate  through  one  or  more  of  the 
involved  agencies  and  end  up  in  a  central  paper  file. 

During  the  March  meeting  the  data  collection  forms  to  be  used  during  the 
OT-III  were  finalized. ^  The  complete  set  of  forms  used  during  the  Ml  OT-III, 
consisting  of  eleven  different  data  collection  instruments  appears  in  Appen¬ 
dices  C  and  D.  The  forms  in  Appendix  C  (Forms  39,  40,  41)  were  designed  by 
ARI  for  the  purpose  of  identifying  training  deficiencies.  The  remaining  forms 
(Appendix  D)  were  to  be  used  by  other  agencies  for  their  own  purposes. 

Diagnostic  Tests 

Members  of  the  TCATA  Fightability  Team  visited  Fort  Knox  on  28  April  1980 
and  met  with  representatives  of  USAARMC,  ARI,  OAFMS,  and  TRASANA.  Among  the 
topics  discussed  were  pre/post  diagnostic  tests,  the  NETT  training  schedule  and 
POI,  data  collection  forms,  the  pilot  test  scheduled  at  Fort  Knox  prior  to  the 
OT-III,  control  of  the  OT-III  data  collection,  and  resource  and  range  require¬ 
ments  associated  with  the  operational  test  (OT) .  Authorization  had  been  ob¬ 
tained  by  TRASANA  to  administer  pre-diagnostic  tests  on  the  M60A1  prior  to 
transition  training  and  post-diagnostic  tests  on  the  Ml  following  transition 
training.  During  the  April  meeting,  a  list  of  tasks  to  be  tested  on  the  pre¬ 
diagnostic  test  and  a  separate  list  to  be  tested  on  the  post -diagnostic  test 
were  compiled  and  approved  by  the  participating  agencies.  It  was  decided  that 
diagnostic  tests  would  be  administered  and  scored  by  instructors  from  the  NETT. 

Scheduling 

Much  time  was  devoted  to  constructing  a  revised  training  schedule  for  the 
OT-III.  This  schedule  established  starting  and  finishing  dates  by  company  and 
MOS  for  individual  and  collective  training  on  the  Ml,  and  for  pre-  and  post¬ 
diagnostic  testing.  The  schedule  called  for  the  pre-diagnostic  for  MOS  45E 
turret  mechanics  in  Company  //I  to  be  given  8-12  September  1980  to  be  followed 
by  the  45E  transition  training  and  the  45E  post-diagnostic.  The  diagnostics 
and  transition  training  for  the  MOS  63E  track  vehicle  mechanics  would  follow 
closely  on  the  heels  of  the  45E  training.  Training  for  the  19L  Ml  drivers  was 
scheduled  to  start  13  November  and  for  the  19K  gunner/loader/tank  commander 
19  November.  A  total  of  four  companies  were  to  be  trained  and  tested  in  this 
way,  with  the  last  company  finishing  their  post-diagnostic  in  mid-April  1981.3 
The  training  and  testing  schedule  was  arranged  at  this  time  because  of  TCATA* s 
requirements  to  schedule  ranges  well  in  advance  and  to  finalize  the  Detailed 
Test  Plan. 


Although  minor  changes  would  be  made  in  some  of  the  forms  subsequent  to 
the  March  meeting,  the  forms  settled  upon  in  this  meeting  were  for  all  prac¬ 
tical  purposes  the  forms  used  during  the  OT-III. 

3 

The  four  company  plan  was  later  dropped  in  favor  of  a  three  company  plan 
due  to  a  slower  than  expected  delivery  schedule  of  the  Ml  tank. 
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Further  Plans 


Plans  on  how  to  organize  the  data  collection  effort  were  further  focused. 
Plans  called  for  the  majority  of  the  data  to  be  collected  by  TCATA  data  collec¬ 
tors,  with  OAFMS  personnel  functioning  as  quality  control  monitors.  Personnel 
from  DTD,  ARI-Fort  Knox,  TRASANA  and  OAFMS  would  function  as  data  analysts  and 
as  data  collectors  on  an  as  needed  basis.  Plans  were  also  made  to  have  a  data 
assignment  roster  for  all  training  events  so  that  each  person  collecting  data 
would  know  which  event  he/ she  was  to  observe,  what  forms  to  use,  and  where  the 
event  would  occur,  at  least  24  hours  prior  to  the  event. 

Pilot  Test  of  the  OT-III  Data  Collection  Forms 

At  the  28  April  meeting,  the  upcoming  Ml  RAM  test  to  be  conducted  19-23 
May  1980  was  also  discussed.  It  was  decided  that  this  test  should  definitely 
be  used  as  a  vehicle  for  piloting  all  training  data  collection  forms  to  be 
used  during  the  OT-III. 

As  planned,  data  collection  forms  to  be  used  during  the  OT-III  were 
piloted  in  conjunction  with  the  Ml  RAM  test  conducted  at  Fort  Knox  in  May  1980. 
OAFMS  scheduled  and  controlled  the  data  collection  activities.  OAFMS  was  as¬ 
sisted  in  collecting  data  by  DTD,  TCATA,  and  ARI-Fort  Knox.  Completed  forms 
were  returned  by  all  agencies  to  OAFMS.  No  problems  were  encountered  in  using 
the  data  collection  forms  during  the  pilot  test;  therefore  no  significant 
changes  were  made  in  the  forms  as  a  result  of  the  pilot  test. 

TCATA' s  Detailed  Test  Plan 


In  an  August  1980  meeting  at  Fort  Hood,  ARI  got  its  first  look  at  TCATA1 s 
Detailed  Test  Plan  for  the  Ml  OT-III.  The  Detailed  Test  Plan  specified  what 
data  would  be  collected,  who  would  collect  it,  and  how  the  data  would  be  dis¬ 
tributed  once  it  was  collected.  The  Detailed  Test  Plan  provided  for  collecting 
training  data  and  test  data  for  each  event  occurring  during  transition  train¬ 
ing.  The  kinds  of  training  data  to  be  collected  can  best  be  seen  by  referring 
to  the  data  collection  forms  in  Appendices  C  and  D.  Each  of  the  training  data 
collection  forms  was  assigned  a  number  for  ease  of  reference.  Two  kinds  of 
test  data  were  collected  —  individual  performance  data  on  end-of-block  tests 
and  diagnostic  test  data. 

A  data  collection  team,  composed  of  a  Company  Team  Chief  Data  Collector, 
three  Platoon  Team  Chief  Data  Collectors,  and  one  Tank  Data  Collector  per  tank, 
was  to  collect  the  MOS  45E,  MOS  63E  and  MOS  19K/L  training  data  for  each  com¬ 
pany.  The  tank  data  collector  would  complete  Form  38a  (Personnel  Status  Re¬ 
port)  and  Form  40  (Observation  of  Test  Events).  The  platoon  data  collector 
would  complete  Form  37  (Training  Aids  Data  Sheet),  Form  39  (Observation  of 
Training),  Form  41  (Training  Environment),  Form  42  (Student  Questionnaire), 
and  Form  44  (Instructor  Questionnaire).  The  company  chief  data  collector 
would  supervise  the  other  data  collectors,  review  and  consolidate  Forms  37, 

38a,  39,  40,  41,  42  and  44,  and  return  these  to  Data  Control  at  the  Field  Test 
Center.  At  Data  Control,  the  forms  would  be  reproduced  and  copies  distributed 
to  appropriate  agencies  (e.g. ,  ARI,  OAFMS)  for  analysis.  The  original  of  each 
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form  would  go  to  TCATA's  Data  Reduction  Branch  where  the  data  would  be  entered 
into  TCATA's  computer  for  additional  analysis  and  the  original  raw  data  would 
be  placed  in  a  master  file  for  future  reference.  Score  sheets  indicating  end- 
of-block  test  performance  of  each  individual  soldier  for  each  block  of  instruc¬ 
tion  would  be  completed  by  NET  team  instructors.  Completed  score  sheets  would 
be  given  to  the  Company  Team  Chief  Data  Collector,  who  would  forward  them  to 
Data  Control  for  reproduction  and  distribution  to  the  various  agencies.  In 
addition  to  end-of-block  tests,  diagnostic  tests  measuring  individual  perform¬ 
ance  levels  just  prior  to  and  following  individual  training  on  the  Ml  tank 
were  to  be  administered  to  100%  of  the  tank  crewmen  and  mechanics  participating 
in  the  operational  test.  Pre-diagnostic  and  post-diagnostic  test  data  were  to 
be  handled  in  the  same  way  as  the  end-of-block  scoresheets.  NETT  instructors 
were  to  collect  the  data.  The  diagnostic  test  data  would  then  be  passed  along 
to  the  Company  Team  Chief  Data  Collector,  who  would  forward  the  data  to  Data 
Control  for  reproduction  and  distribution  to  participating  agencies. 


SECTION  IV 


EVALUATING  THE  Ml  TRANSITION  TRAINING  LESSON  PLANS 


Obtaining  the  Lesson  Plans 

As  part  of  the  evaluation  of  the  transition  training  program  given  by  the 
NETT  during  the  OT-III,  ARI  was  to  evaluate  the  NET  program  as  detailed  in  the 
lesson  plans.  The  lesson  plans  were  also  needed  to  prepare  some  of  the  obser¬ 
vation  forms  to  be  used  during  the  evaluation.  In  a  memorandum  dated  20  Feb 
79,  ARI  formally  requested  that  the  NETT  provide  ARI  with  copies  of  any  MOS 
19K/L  training  materials  developed  for  use  during  the  Ml  OT-III.  ARI  obtained 
a  set  of  the  MOS  19K/L  lesson  plans  from  OAFMS. 

The  original  transition  training  materials  did  not  provide  useful  training 
prescriptions  and  contained  omissions.  Some  of  the  tasks  to  be  trained  during 
the  Ml  OT-III  were  not  covered  by  the  lesson  plans.  Though  there  were  short¬ 
comings  in  the  original  version  of  the  Ml  OT-III  transition  training  lesson 
plans,  the  NETT  indicated  that  a  revised  set  of  lesson  plans  would  be  forth¬ 
coming  shortly. 

Review  and  Initial  Evaluation  of  Lesson  Plans 


The  second  version  of  the  MOS  19K/L  lesson  plans  was  obtained  from  OAFMS 
in  April  1980.  OAFMS  had  reviewed  these  lesson  plans  and  had  suggested 
changes,  some  of  which  were  incorporated  into  the  lesson  plans  by  the  NETT  and 
some  of  which  were  not.  A  cursory  review  of  the  lesson  plans  by  ARI  indicated 
that  many  of  the  problems  noted  by  ARI  in  the  original  version  had  not  been 
addressed.  In  discussing  these  problems  with  OAFMS,  ARI  decided  that  a  thorough 
review  of  the  lesson  plans  was  called  for.  Each  lesson  plan  needed  to  be  eval¬ 
uated  to  ensure  that  it  contained  the  necessary  information  and  was  internally 
consistent.  In  general  a  lesson  plan  should  include  detailed  information  on 
what  is  to  be  taught  and  tested  and  how  it  is  to  be  taught  and  tested.  With 
these  ideas  in  mind,  ARI  proceeded  to  evaluate  each  lesson  plan,  in  turn, 
noting  any  problems  that  might  require  corrective  action. 

The  evaluation  quickly  bogged  down  because  of  the  tedious  nature  of  the 
task.  Each  lesson  plan  seemed  to  come  with  its  own  unique  set  of  problems, 
i  and  each  separate  problem  seemed  to  require  a  different  corrective  action. 

This  made  it  difficult  to  discern  where  the  evaluation  for  a  particular  lesson 
plan  should  begin  and  where  it  should  end.  It  soon  became  clear  that  a  more 
systematic  approach  to  the  task  was  needed. 

A  Systematic  Approach  to  Lesson  Plan  Evaluation 

A  set  of  criteria  for  evaluating  lesson  plans  was  developed  and  applied 
to  the  MOS  19  K/L  lesson  plans.  The  use  of  these  criteria  greatly  simplified 
the  task  of  evaluating  the  lesson  plans.  By  using  the  criteria,  the  evaluator 
was  able  to  focus  on  one  aspect  of  the  lesson  plan  at  a  time,  while  ensuring 
that  all  important  aspects  of  the  lesson  plan  were  evaluated.  The  criteria  de¬ 
veloped  for  evaluating  the  MOS  19  K/L  lesson  plans  are  listed  in  Appendix  E. 
Criteria  are  listed  for  evaluating  training  objectives,  training  procedures, 
and  end-of-block  tests. 


The  evaluation  of  the  MOS  19  K/L  lesson  plans  produced  43  pages  of  com¬ 
ments  and  suggestions  regarding  lesson  plan  deficiencies.  ARl’s  comments  and 
suggestions  concerning  the  lesson  plans  were  attached  to  a  memorandum  dated 
1  August  1980  and  sent  to  the  Director  of  Training  Developments  (DTD).  Follow¬ 
up  contacts  with  DTD  indicated  that  DTD  had  received  the  lesson  plan  evalua¬ 
tion,  but  that  the  evaluation  had  not  filtered  down  to  the  personnel  who 
developed  the  lesson  plans  and  who  were  responsible  for  making  any  changes  in 
the  lesson  plans.  It  was  not  clear  whether  any  of  ARl’s  comments  or  sugges¬ 
tions  had  been  acted  upon.  66 

Revised  Lesson  Plans 

In  September  1980,  DTI)  released  the  MOS  19  K/L  lesson  plans  that  were  to 
be  used  in  training  the  first  company  during  the  Ml  OT-III.  ARI  obtained 
copies  from  the  NETT  after  learning  from  OAFMS  that  the  lesson  plans  had  been 
completed.  This  version  of  the  MOS  19  K/L  lesson  plans  was  more  complete  than 
previous  versions  and  had  eliminated  many  of  the  inconsistencies  identified 
during  ARI  s  evaluation. 4  However  many  of  the  problems  identified  in  the  eval¬ 
uation  still  remained.  Many  of  the  training  objectives  did  not  accurately 
specify  what  the  soldiers  would  be  trained  to  do.  Much  of  the  guidance  pro¬ 
vided  to  the  instructors  on  how  the  training  was  to  be  conducted  was  vague,  and 
many  of  the  tests  designed  to  measure  the  soldier’s  ability  to  perform  the  tasks 
in  the  training  objectives  did  not  mirror  the  requirements  of  those  objectives. 


It  is  not  at  all  clear  whether  the  elimination  of  the  inconsistencies 
resulted  from  ARI  s  evaluation  of  the  lesson  plans  or  were  the  result  of  DTD 
own  internal  review  process. 
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FOR  THE  Ml  OT-III  EVALUATION 


The  August  1980  meeting  was  the  final  planning  meeting  prior  to  the  begin¬ 
ning  of  the  Ml  OT-III.  At  this  point  most  of  the  details  of  the  operational 
test  had  been  worked  out.  The  training  schedule  was  firm  given  that  tanks 
were  delivered  on  time,  ranges  had  been  reserved  for  the  test,  the  detailed 
plan  for  collecting  the  data  had  been  developed,  responsibilities  of  the  various 
participating  agencies  had  been  established,  the  data  collection  forms  had  been 
finalized,  and  personnel  for  collecting  and  analyzing  the  data  had  been  com¬ 
mitted  to  the  task.  Workshops  had  also  been  scheduled  for  training  the  Ml 
OT-III  data  collectors. 

All  data  collectors  participating  in  the  Ml  OT-III  were  required  to  com¬ 
plete  a  workshop  in  order  to  be  trained  on  data  collecting  procedures  and  to 
become  familiar  with  the  training  evaluation  forms.  In  contrast  to  previous 
workshops,  very  little  time  would  be  spent  on  the  planning  and  analysis  aspects 
of  training  program  evaluation.  Planning  the  TPE  and  analyzing  the  data  col¬ 
lected  would  only  be  discussed  briefly  in  order  to  help  the  data  collectors 
understand  the  reasons  for  collecting  the  data.  Instead  of  the  18  worksheets 
from  the  Harless  Guidelines  covered  in  earlier  workshops,  only  three  data  col¬ 
lection  forms  (39,  AO,  &  41)  would  be  covered.  ^  Whereas  the  Harless  Guidelines 
were  used  as  both  a  training  aid  and  a  job  aid  in  earlier  workshops,  the  Guide¬ 
lines  were  not  appropriate  to  use  in  the  OT-III  workshops.  Not  only  did  the 
Harless  Guidelines  provide  far  more  information  than  was  needed,  but  the  Guide¬ 
lines  were  written  for  a  user  having  some  familiarity  with  educational  tech¬ 
nology.  The  soldiers  who  would  collect  the  OT-III  data  were  naive  with  regard 
to  educational  technology. 

What  was  needed  was  a  job  aid  designed  specifically  for  individuals  un¬ 
sophisticated  in  educational  technology  whose  job  it  was  to  collect  training 
evaluation  data.  In  response  to  this  need,  ARI  developed  an  observer's  job 
aid  (Witmer,  Note  3).  The  observer's  job  aid  borrowed  much  from  the  Harless 
Guidelines,  but  the  language  used  was  simplified  with  many  technical  terms 
being  eliminated,  and  the  scope  was  narrowed  to  focus  on  those  activities  per¬ 
formed  by  the  data  collector.  Like  the  Harless  Guidelines,  data  collection 
forms  were  included  in  the  job  aid.  The  job  aid  described  i:ow  the  forms 
should  be  completed,  including  an  explanation  of  each  item  on  the  forms. 

The  forms  described  in  the  observer's  job  aid  were  to  be  used  to  collect 
information  about  the  Ml  transition  training  program.  Data  would  be  collected 
on  the  training  environment,  the  training  process  and  the  testing  process. 

This  data  would  be  used  in  conjunction  with  the  performance  results  on  the 


A  fourth  form  for  determining  if  the  training  given  conforms  to  the 
training  planned  as  described  in  the  lesson  plans  was  scheduled  to  be  covered 
in  the  workshops,  but  because  the  first  group  of  data  collectors  to  use  the 
form  did  not  find  it  useful,  the  form  was  not  covered  in  subsequent  workshops. 


tests  given  at  the  end  of  each  block  of  instruction  to  identify  training-caused 
performance  deficiencies  in  the  Ml  transition  training  program.  The  training 
analyst  (an  ARI  researcher)  would  identify  the  training  deficiencies  and  make 
recommendations  to  the  NETT  on  how  the  training  program  should  be  modified  to 
eliminate  the  deficiencies.  To  assist  the  training  analyst  in  making  modifi¬ 
cations  to  the  training  program  based  on  the  deficiencies  identified,  ARI  de¬ 
veloped  a  job  aid  for  modifying  ineffective  or  inefficient  training  (Kristiansen 
Note  4).  This  training  modifications  job  aid  was  designed  to  be  used  with  the 
observer's  job  aid  in  evaluating  the  Ml  transition  training  program. 6 


In  th.^Lmil0nVn  Pract;ce  T*  feedbaCk  “cre  basea  sl"ilar 
in  the  Seville  product  mentioned  earlier  in  this  paper. 


FIELD  TRIAL  OF  THE  TPE  SYSTEM 
Scope  and  Purpose  of  Field  Trial 

ARI's  purpose  in  evaluating  the  Ml  OT-III  transition  training  program  was 
to  field  test  the  job  aids  and  data  collection  forms  developed  by  ARI  for  evalu¬ 
ating  training  programs.  The  objective  was  not  just  to  determine  if  the  TPE 
methodology  worked,  but  to  demonstrate  that  it  can  be  used  by  Army  personnel 
to  evaluate  the  training  program  for  a  major  developing  weapon  system.  In 
order  to  accomplish  this  objective  mid-level  noncommissioned  officers  were 
trained  to  be  data  collectors.  TPE  workshops  were  held  to  train  turret  mechanic 
(45E),  track  and  vehicle  mechanic  (63E)  and  tank  crew  (19K/L)  data  collectors. 
Separate  workshops  were  conducted  to  train  each  of  three  sets  of  tank  crew  data 
collectors  for  each  of  the  three  companies  undergoing  transition  training.  The 
workshops  were  conducted  for  each  set  of  data  collectors  just  prior  to  their 
participation  in  the  operational  test.  Table  1  lists  the  TPE  training  sessions 
conducted  for  the  Ml  OT-III  data  collectors.  During  the  workshop  each  data  col¬ 
lector  received  a  copy  of  the  observer's  job  aid  for  use  in  the  workshop  and  on 
the  job. 

The  data  collection  team,  consisting  of  a  company  data  collector,  three 
platoon  data  collectors,  and  one  tank  data  collector  per  tank,  was  supervised 
by  the  TCATA  Fightability  Team  (a  designator  for  that  group  in  TCATA  responsible 
for  evaluating  the  NET  program).  The  day-to-day  activities  of  the  tank  data 
collectors  were  controlled  by  the  company  chief  data  collector.  The  TCATA 
Fightability  Team  had  little  interaction  with  the  tank  data  collectors.  Their 
main  contact  with  the  data  collection  team  was  through  the  company  data  col¬ 
lector  and  came  only  when  the  data  forms  reaching  data  control  were  improperly 
completed.  In  addition  to  the  military  chain  of  command  supervision  and  the 
job  aids,  data  collectors  were  periodically  contacted  by  the  ARI  data  analyst 
or  by  OAFMS  personnel  to  assist  them  with  any  problems  they  might  be  having  in 
completing  the  evaluation  forms. 

TABLE  1.  TPE  Training  Sessions  Conducted  for  Ml  OT-III  Data  Collectors 


Workshop  Training 
Dates 

Workshop  Participant's 
Specialty 

Number  of 

Soldiers  Trained 

16-18  Sep  80 

turret  mechanic 

6 

8-10  Oct  80 

track  vehicle  or 
turret  mechanic 

15 

21-23  Oct  80 

tank  crewman 

20 

14  Nov  80 

tank  crewman 

5 

1-3  Dec  80 

tank  crewman 

17 

23-26  Jan  81 

tank  crewman 

17 

For  purposes  of  field  testing  the  TPE  methodology  and  job  aids,  ARI  was 
concerned  only  with  phase  1  (transition  training)  of  the  Ml  OT-III.  Phase  I 
began  with  the  administration  of  the  prediagnostic  test.  The  prediagnostic 
test  was  designed  to  determine  if  the  soldiers  to  be  trained  on  the  Ml 
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possessed  all  of  the  prerequisite  skills  on  the  M60A1  prior  to  undergoing  tran¬ 
sition  training.  Following  the  prediagnostic,  soldiers  were  given  individual 
training  on  the  Ml  tank.  After  individual  training,  mechanics  were  given  a 
post-diagnostic  test.  For  tank  crewmen,  individual  training  was  followed  by 
collective  training  which  in  turn  was  followed  by  the  post-diagnostic  test. 

The  post-diagnostic  test  consisted  of  the  same  tasks  on  the  Ml  that  were  tested 
on  the  M60A1  during  the  prediagnostic  test  plus  some  additional  tasks  selected 
from  the  transition  training  program.  The  tank  turret  mechanics  (MOS  45E)  were 
the  first  to  be  given  the  prediagnostic  and  to  undergo  transition  training. 
Individual  training  for  the  turret  mechanics  began  in  late  September  1980. 

Next  to  be  trained  on  the  Ml  tank  were  the  track/vehicle  mechanics  (MOS  63E) ; 
individual  training  for  these  mechanics  began  in  mid-October.  Three  tank  com¬ 
panies  received  individual  and  collective  tank  crew  training  in  successive 
presentations  of  the  MOS  19K/L  transition  training  program.  Changes  were  made 
following  each  successive  presentation  of  the  transition  training  so  that  the 
training  received  by  the  second  company  was  not  identical  to  that  received  by 
the  first  company.  And  the  third  company's  training  differed  from  both  of  the 
companies  preceding  it.  Changes  were  not  extensive,  however,  and  did  not  in¬ 
volve  changes  in  time  or  resource  allocation.  The  training  schedule  for  each 
of  the  three  tank  companies  is  listed  in  Table  2. 

TABLE  2.  Phase  1,  Ml  OT-III  MOS  19K/L  Training  Dates 


Event 

Dates 

co  n 

Co  it  2 

Co  it  3 

Individual  Training  Begins 
Individual  Training  Ends 
Collective  Training  Begins 
Collective  Training  Ends 

12  Nov  80 

3  Dec  80 

4  Dec  80 

20  Dec  80 

5  Jan  81 

22  Jan  81 

23  Jan  81 

6  Feb  81 

10  Feb  81 

27  Feb  81 

28  Feb  81 
12  Mar  81 

ARI-trained  data  collectors  completed  TPE  data  collection  forms  during 
the  training  of  45E  and  63E  mechanics,  as  well  as  during  19K/L  tank  crewman 
training  for  each  of  three  companies.  During  the  19K/L  training,  an  ARI  re¬ 
searcher  was  present  on  site  to  analyze  the  19K/L  data  as  they  were  collected. 
Because  ARl's  time  on  site  was  limited  and  because  the  data  available  from  the 
19K/L  training  were  considered  adequate  for  testing  the  usefulness  of  TPE,  ARI 
chose  not  to  analyze  the  45E  and  63E  data.  In  accordance  with  an  agreement 
made  by  ARI  with  OAFMS  and  TCATA  prior  to  the  beginning  of  the  operational 
test,  OAFMS  and  TCATA  utilized  the  45E  and  63E  data  in  satisfying  their  respec 
tive  missions.  Examination  of  completed  TPE  data  collection  forms  from  the 
mechanics  training  courses  and  feedback  from  OAFMS  and  TCATA  indicated  that 
the  completed  TPE  data  collection  forms  provided  consider. ;ble  information  re¬ 
garding  mechanics  training. 

As  input  to  the  analysis  of  the  effectiveness  of  the  19K/L  training,  ARI 
used  TPE  Forms  39,  40,  and  41  (see  Appendix  C)  completed  by  the  tank  data  col¬ 
lectors,  the  19K/L  lesson  plans  and  the  scoresheets  from  each  19K/L  block  of 


instruction.  Additional  input  to  the  analysis  came  from  ARl's  own  observation 
of  some  of  the  19K/L  transition  training  and  through  observations  made  by 
representatives  of  some  of  the  other  agencies  involved  in  the  Ml  operational 
test.  Neither  data  on  mechanics  (45E,  63E)  training  nor  diagnostic  test  re¬ 
sults  were  used  as  input  to  the  TPE  analysis. 

Conducting  the  Field  Trial 

The  field  test  of  the  TPE  materials  was  performed  in  conjunction  with  pro¬ 
viding  feedback  to  the  NET  team  on  the  effectiveness  of  the  Ml  transition 
training.  The  method  used  for  the  field  test  basically  consisted  of  collectirg 
training  and  performance  data,  summarizing  and  analyzing  that  data,  making 
recommendations  for  changing  the  training  based  on  the  analysis,  and  determin¬ 
ing  the  effects  of  the  changes  made.  The  procedural  steps  used  in  field  test¬ 
ing  the  TPE  materials  are  outlined  below.  The  steps  outlined  were  repeated  for 
each  lesson  or  block  of  instruction  in  the  individual  training  portion  of  the 
MOS  19K/L  transition  training  program. 

Data  on  the  training  environment  (Form  41),  traihing  process  (Form  39), 
and  testing  process  (Form  40)  were  collected  by  ARI-trained  NCO  tank  data  col¬ 
lectors.  ^  Completed  forms  were  checked  by  the  company  chief  data  collector  for 
omissions  or  inconsistencies  and  then  returned  to  the  Field  Test  Center  for 
copying  and  distribution.  Data  on  end-of-block  individual  test  performance 
were  recorded  on  score  sheets  by  NETT  instructors  and  given  to  the  company 
chief  data  collector.  The  company  chief  data  collector  forwarded  the  original 
score  sheets  to  the  Field  Test  Center  for  reproduction  and  distribution. 

At  the  Field  Test  Center  the  ARI  training  analyst  received  copies  of  com¬ 
pleted  Forms  39,  40,  and  41  and  the  score  sheets  as  they  came  back  to  the  Field 
Test  Center.  For  training  conducted  on  the  tank,  the  ARI  analyst  received  data 
collection  Forms  39,  40,  and  41  from  each  of  the  13  tanks  (the  number  of  tanks 
in  each  company  and,  hence,  the  number  of  training  sites,  or  stations,  per  com¬ 
pany).  For  classroom  training,  fewer  forms  were  received  for  each  block  of 
instruction. 

For  each  block  of  instruction,  the  analyst  summarized  the  data  appearing 
on  the  data  collection  forms.  All  the  data  recorded  on  Form  39  for  a  given 
block  of  instruction  were  combined  on  a  single  summary  data  worksheet.  Simi¬ 
larly  the  training  analyst  prepared  summary  data  worksheets  for  Forr.;’*  40  and 
41.  The  scoresheets  for  each  block  of  instruction  were  reviewed  to  obtain  the 
percentage  of  soldiers  receiving  NO-GO's  for  each  task  and  subtask.  Tasks  or 
subtasks  for  which  20%  or  more  of  the  soldiers  tested  received  a  NO-GO  on  their 
first  trial  were  considered  to  represent  performance  deficiencies  (the  standard 
provided  by  USAARMC) .  Possible  causes  of  these  deficiencies  were  identified 
from  the  training  and  testing  data  recorded  on  Forms  39,  40,  and  41.  From 
these  causes,  the  analyst,  with  the  help  of  the  modifications  job  aid,  sug¬ 
gested  changes  to  the  block  of  instruction  for  eliminating  the  performance 
deficiencies. 

^This  departure  from  the  procedure  that  was  planned  has  the  advantage  of 
providing  training  data  from  each  training  station  (i.e.,  each  tank). 


For  some  blocks  of  Instruction,  review  of  the  scoresheets  did  not  turn  up 
any  performance  deficiencies.  Nevertheless  the  testing  observation  form  (Form 
40)  was  reviewed  to  determine  if  there  were  any  irregularities  in  the  testing 
procedures.  If  any  problems  were  observed  in  the  testing  process  that  could 
affect  the  validity  of  the  test  results,  changes  were  suggested  to  the  testing 
process  based  on  guidance  provided  in  the  modifications  job  aid. 

The  training  analyst  summarized  his  findings  separately  for  each  block  of 
instruction  in  a  memorandum  to  DTD.  The  memorandum  identified  the  tasks  and 
subtasks  for  which  performance  deficiencies  were  found  and  specified  changes  to 
the  training  program  to  correct  the  deficiencies.  The  memorandum  also  identi¬ 
fied  problems  with  the  testing  procedures  and  suggested  changes  as  appropriate. 
Memoranda  were  usually  forwarded  to  DTD  within  one  day  of  the  data  becoming 
available  for  analysis. 

Changes  in  the  transition  training  program  made  by  the  NETT  were  subject 
to  the  approval  of  DTD.  Some  of  the  changes  were  generated  by  the  NETT  itself 
and  others  were  suggested  by  DTD  and  OAFMS.  Changes  suggested  by  ARI  were  in¬ 
corporated  into  the  transition  training  program  through  DTD  since  ARI  was  not 
allowed,  by  agreement  with  DTD,  to  interface  directly  with  the  NETT.  ARI  was 
informed  about  changes  made  by  the  NET  team  in  the  transition  training  POI 
by  DTD. 

Field  Trial  Assessment  of  the  TPE  System 

To  determine  if  changes  made  in  the  transition  training  POI  had  an  effect 
on  the  quality  of  training  and  test  performance,  the  data  obtained  before 
changes  were  made  were  compared  to  the  data  obtained  subsequent  to  the  changes 
being  made.  For  the  Ml  OT-III  this  entailed  comparing  the  data  obtained  for 
each  successive  iteration  of  training  (i.e..  Company  1  data  were  compared  to 
Company  2,  and  Company  2  data  were  compared  to  Company  3).  Two  kinds  of  data 
were  compared.  The  NO-GO  rates  for  performance  on  the  test  given  after  each 
block  of  instruction  were  compared  for  each  task.  Data  collected  during  train¬ 
ing  and  testing  using  Forms  39,  40,  and  41  were  compared  to  determine  if  the 
changes  made  in  the  transition  training  were  reflected  in  the  data  recorded  on 
the  evaluation  forms.  Specifically,  the  number  of  comments  made  for  each  item 
on  the  forms  were  compared  from  one  company  to  the  next. 8 

The  field  trial  was  conducted  in  order  to  answer  a  number  of  questions  re¬ 
garding  the  usefulness  and  effectiveness  of  the  TPE  system.  Among  the  questions 
to  be  answered  were  the  following: 

Can  the  TPE  methodology  be  used  by  the  Army  to  evaluate  the  training 
program  developed  for  a  major  weapons  system? 


Comments  recorded  on  the  evaluation  forms  generally  refer  to  training  or 
testing  problems  and  thus  may  serve  as  a  useful  index  of  training  effectiveness. 
The  number  and  quality  of  comments  also  serves  as  a  rough  measure  of  data  col¬ 
lector  proficiency. 


.  Can  mid-level  NCO's  be  trained  to  collect  evaluation  data  given  only  a 
short  workshop  in  TPE  methodology  and  the  guidance  provided  by  the  ob¬ 
server's  job  aid? 

.  Can  a  training  analyst  familiar  with  the  TPE  methodology  identify 
training  program  deficiencies  and  their  causes  from  the  data  recorded 
on  the  TPE  forms  by  NCO's? 

.  Can  the  training  analyst  make  recommendations  for  modifying  the  train¬ 
ing  program  from  the  deficiencies  identified  that  are  both  convincing 
and  in  a  form  that  can  be  used  by  the  training  developer?  Is  the  modi¬ 
fications  job  aid  useful  in  specifying  these  recommendations? 

.  When  changes  are  made  in  the  training  program,  is  the  TPE  methodology 
sensitive  enough  to  detect  these  changes? 

.  Do  changes  suggested  through  the  TPE  process  reduce  the  NO-GO  rate  on 
the  tests  given  at  the  end  of  each  block  of  instruction? 

.  How  should  the  TPE  system,  including  the  data  collection  forms  and  job 
aids,  be  altered  in  order  to  increase  the  usefulness  and  effectiveness 
of  the  system  as  an  evaluation  tool? 

The  procedures  used  in  collecting  data  pertaining  to  each  question  will 
be  described.  The  data  collected  will  be  reported  and  the  conclusions  drawn 
from  the  data  will  be  discussed.  It  should  be  noted  that  the  types  of  data 
that  could  be  collected  during  the  field  test  were  often  limited  by  externally 
imposed  constraints,  forcing  ARI  to  rely  to  some  extent  on  indirect  evidence. 
Because  evidence  was  lacking  in  some  cases,  not  all  the  questions  posed  were 
answered  conclusively. 

The  ability  of  the  Army  to  use  the  TPE  system  in  evaluating  the  training 
program  for  a  major  weapons  system  was  the  central  issue  to  be  resolved  during 
the  Ml  OT-III.  The  Army's  need  for  an  evaluation  methodology  such  as  the  TPE 
system  had  been  established  previously  as  evidenced  by  the  interest  that  many 
organizations  had  shown  in  using  the  system.  With  ARl's  assistance  these  or¬ 
ganizations  had  used  the  TPE  data  collection  forms  in  evaluating  training  pro¬ 
grams  with  some  success.  However,  the  programs  to  which  the  TPE  system  was 
applied  prior  to  the  Ml  OT-III  were  much  narrower  in  scope,  and  because  the 
purposes  of  these  earlier  evaluations  were  limited,  the  TPE  system  was  not 
fully  exercised.  The  first  opportunity  to  assess  applicability  of  the  TPE 
system  to  the  training  program  for  a  major  weapons  system  using  Army  personnel 
came  during  the  Ml  OT-III. 

Information  was  drawn  from  several  sources  in  assessing  the  Army's  ability 
to  apply  the  TPE  system  to  the  evaluation  of  the  OT-III  transition  training 
program.  One  source  of  information  was  ARl's  observations  regarding  the  plan¬ 
ning  process  that  preceded  the  Ml  OT-III.  As  described  before,  the  scope  of 
the  OT  required  extensive  coordinated  planning  by  several  organizations  includ¬ 
ing  ARI.  ARl's  role  in  the  planning  process  was  to  ensure  that  the  TPE  system 
was  incorporated  into  the  overall  test  plan  in  such  a  manner  as  to  fully  utilize 
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its  potential  as  an  evaluation  instrument  given  only  those  Army  personnel  and 
resources  that  could  be  made  available  for  the  evaluation  of  the  training  pro¬ 
gram.  TCATA,  who  controlled  the  data  collection  effort,  was  able  to  procure 
the  personnel  and  resources  required  to  utilize  the  TPE  system,  and  the  TPE 
system  was  incorporated  into  the  detailed  test  plan  in  a  way  that  was  acceptable 
to  ARI  and  all  of  the  other  agencies  involved  in  the  program  evaluation.  This 
demonstrated  that  the  Army  could  incorporate  the  TPE  system  in  an  overall  tes’z 
plan  for  evaluating  the  training  program  for  a  major  weapons  system  without 
compromising  the  TPE  system  or  the  overall  test  of  which  it  is  a  part. 

A  second  source  of  information  regarding  the  Army’s  ability  to  use  the 
TPE  system  was  the  Army  implementation  of  the  TPE  system  during  the  OT.  Among 
the  important  factors  in  the  implementation  were:  the  Army’s  capability  to 
follow  their  own  plans  in  using  the  system;  the  timeliness  with  which  the 
evaluation  data  were  collected,  forwarded  to  the  training  analyst,  and  acted 
upon  in  accordance  with  the  analyst's  recommendations;  the  quantity  and  quality 
of  the  data  collected  by  the  NCO's  using  the  TPE  worksheets;  and  the  extent  to 
which  the  data  were  used  by  the  various  organizations. 

The  manner  in  which  the  TPE  system  was  used  was  in  accordance  with  the 
detailed  test  plan  with  two  notable  exceptions.  The  task  of  collecting  the 
TPE  data  was  not  shared  by  the  platoon  and  tank  data  collectors  as  planned. 
Rather  the  tank  data  collectors  were  required  to  shoulder  almost  all  of  the 
data  collection  task,  including  certain  data  collection  requirements  that  were 
added  after  the  0T-II1  began.  Some  evidence  suggests  that  the  performance  of 
the  tank  data  collectors  was  adversely  affected  by  the  large  number  of  data 
collection  forms  they  were  required  to  complete.  Independent  observation  of 
some  blocks  of  instruction  by  the  ARI  analyst  and  evaluators  from  other  organi¬ 
zations  indicate  that  the  quantity  and  quality  of  the  data  collected  by  some 
of  the  data  collectors  was  less  than  might  reasonably  be  expected.  Informal 
conversations  with  some  of  the  data  collectors  also  suggested  that  the  number 
of  data  collection  instruments  to  be  completed  had  a  negative  impact  on  the 
motivation  of  the  soldiers  to  conscientiously  record  their  observations  on  the 
TPE  data  collection  forms.  The  problem  was  exacerbated  by  the  conditions  under 
which  the  data  collectors  were  forced  to  work.  Data  collectors  frequently 
were  required  to  collect  data  all  day  in  wet  or  cold  and  windy  weather.  Often 
they  were  called  upon  to  collect  data  right  through  (and  long  after)  normal 
meal  hours.  Many  of  the  "creature  comforts"  provided  to  the  participating 
units  were  not  given  to  the  data  collectors,  and  they  were  treated  as  unneces¬ 
sary  by  the  NETT  and  by  the  units  being  trained.  A  second  deviation  from  the 
detailed  test  plan  that  adversely  affected  the  evaluation  effort  was  that 
changes  to  the  training  program  were  not  made  on  a  day-to-day  basis  as  planned. 
Plans  called  for  recommendations  for  changes  in  the  training  program  to  be  re¬ 
ceived  by  the  persons  responsible  for  making  those  changes  within  24  hours  of 
the  time  that  the  class  being  evaluated  was  conducted.  Because  of  problems 
unique  to  the  Ml  OT-III  at  Fort  Hood  ARI  did  not  receive  the  data  collection 
forms  in  a  timely  manner  and  the  persons  responsible  for  making  the  changes 
did  not  receive  ARl’s  recommendations  for  changes  until  weeks  after  the  train¬ 
ing  had  occurred.  Upon  receipt  of  ARl's  recommendations,  the  NETT  made  some  of 
the  changes  suggested  by  ARI  and  did  not  make  others.  Unfortunately,  the  NETT 
did  not  document  those  changes  that  were  made  in  the  training  program,  thereby 
leaving  some  question  as  to  the  extent  of  the  changes  made. 
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Despite  some  difficulties  in  using  the  TPE  system,  all  of  the  Army  agen¬ 
cies  involved  in  the  evaluation  of  the  Ml  training  program  used  the  data  gener¬ 
ated  by  the  TPE  system  for  their  respective  purposes.  The  primary  users  were 
OAFMS,  TCATA,  and  ARI.  OAFMS  used  the  data  for  certifying  the  readiness  of 
the  OT-III  players  as  Ml  qualified  crewmen  and  mechanics.  OAFMS  also  used  the 
data  in  certifying  the  effectiveness  of  the  Ml  transition  training  package. 

TCATA  kept  detailed  records  of  all  the  TPE  training  data  and  used  the  data  and 
memoranda  generated  by  ARI  as  input  for  their  own  independent  training  effec¬ 
tiveness  analysis.  DTD  and  TRASANA  also  used  the  data  in  conducting  a  Cost 
and  Training  Effectiveness  Analysis  (CTEA)  on  the  Ml  transition  training  pro¬ 
gram,  although  TRASANA  relied  heavily  on  pre-  and  post-diagnostic  test  data  in 
its  analysis.  Many  of  DTD's  directives  to  the  NETT  to  modify  the  training 
program  were  based  on  changes  first  recommended  by  ARI.  All  of  the  organiza¬ 
tions  involved  in  evaluating  the  effectiveness  of  Ml  transition  training  seemed 
to  find  the  data  useful.  This  is  not  surprising  in  that  the  TPE  system  pro¬ 
vided  the  largest  pool  of  objective  training  data  that  was  available  during 
the  Ml  OT-III. 

The  TPE  system  was  designed  to  be  used  by  soldiers  who  were  not  sophisti¬ 
cated  in  educational  technology  and  who  had  no  previous  experience  in  collect¬ 
ing  or  analyzing  training  data.  The  NCO's  who  served  as  the  tank  data  collec¬ 
tors  and  their  supervisors,  having  neither  methodological  sophistication  nor 
previous  experience,  were  an  ideal  group  for  testing  whether  typical  Army  per¬ 
sonnel  could  collect  evaluation  data  given  a  short  workshop  in  using  the  TPE 
data  collection  forms  and  the  observer's  job  aid.  The  TPE  workshops  conducted 
by  the  training  analyst  provided  the  first  opportunity  for  determining  the 
ability  of  the  data  collectors  to  use  the  TPE  system.  As  a  whole,  the  group 
responded  well  to  the  training  given  in  the  workshops.  Questions  asked  during 
training,  performance  during  the  practice  session,  and  comments  made  during 
the  discussion  following  practice  indicated  a  high  level  of  understanding  for 
the  majority  of  the  soldiers  being  trained.  While  some  of  the  soldiers'  ques¬ 
tions  indicated  misunderstanding  of  a  few  of  the  TPE  terms  used  (e.g.,  isolated 
practice,  level  of  reality),  the  preponderance  of  the  soldiers'  responses  sug¬ 
gested  the  soldiers  could  use  the  TPE  data  collection  forms  to  make  objective 
observations. 

The  quality  and  quantity  of  the  observations  recorded  on  the  forms  during 
the  transition  training,  however,  were  not  what  might  be  expected  on  the  basis 
of  the  performance  demonstrated  during  the  workshop.  Many  of  data  collectors 
were  not  using  the  forms  as  they  were  designed  to  be  used.  The  forms  listed 
specific  items  (see  Appendix  C)  which  required  the  data  collectors  to  observe 
training  to  determine  if  it  met  specific  criteria  described  in  the  observer’s 
job  aid  and  discussed  in  the  TPE  workshop.  When  these  criteria  were  not  met, 
the  data  collectors  were  encouraged  to  record  a  comment  describing  what  went 
wrong.  But  as  the  data  began  to  come  in,  it  became  apparent  that  many  of  the 
data  collectors  were  treating  the  TPE  data  collection  forms  as  simple  check¬ 
lists,  responding  to  the  items  subjectively  based  on  their  general  impressions 
of  the  items  rather  than  using  the  objective  criteria  specified  in  the  observer's 
job  aid  (Witmer,  Note  3).  Furthermore  the  number  of  comments  recorded  on  the 
forms  were  far  fewer  than  might  be  expected  based  on  independent  observations 
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of  the  training  by  the  TPE  analyst  and  others.  Fortunately,  sufficient  data 
were  obtained,  through  the  efforts  of  a  handful  of  data  collectors,  to  allow 
the  transition  training  program  to  be  evaluated  as  planned. 


An  ARI  researcher  analyzed  the  data  provided  by  the  team  of  data  collec¬ 
tors.  Some  of  the  procedures  used  by  the  analyst  were  designed  specifically 
for  evaluating  training  during  the  Ml  OT-III  and  were  undergoing  their  first 
real  test  during  the  OT.  There  was  a  question  as  to  whether  the  training 
analyst  could  identify  training  program  deficiencies  and  their  causes  given 
only  the  data  recorded  by  the  NCO  data  collectors  on  the  data  collection  forms. 
For  the  most  part,  the  analyst  was  able  to  identify  performance  deficiencies 
and  their  causes  from  the  data  recorded  on  the  data  collection  forms.  In  a  few 
cases  the  analyst  was  aided  in  identifying  causes  by  his  own  independent  ob¬ 
servations  of  the  training  and  by  data  from  other  sources.  There  was  at  least 
one  class  for  which  the  data  recorded  on  the  TPE  forms  provided  no  clues  re¬ 
garding  the  causes  of  the  performance  deficiencies  observed.  In  this  particu¬ 
lar  case,  it  was  necessary  for  the  analyst  to  review  some  additional  data  col¬ 
lected  by  TCATA  and  to  question  some  of  the  persons  who  had  observed  the  class 
about  how  the  class  was  conducted  in  order  to  identify  the  cause  of  the  defi¬ 
ciency.  It  should  be  stressed,  however,  that  the  analyst  was  forced  to  resort 
to  search  for  the  necessary  information  only  because  the  data  collectors  did 
not  record  the  necessary  information  on  the  TPE  forms,  and  was  not  due  to  the 
design  of  the  forms  themselves. 

Having  identified  training  deficiencies  and  their  causes,  the  analyst  was 
in  the  position  to  recommend  changes  to  the  transition  training  to  correct  the 
deficiencies.  The  analyst  derived  recommendations  for  changing  the  training 
program  from  the  modifications  job  aid  (Kristiansen,  Note  4).  The  modifica¬ 
tions  job  aid  was  used  by  the  analyst  to  specify  what  the  NETT  should  do  to 
correct  each  of  the  performance  deficiencies  observed.  The  NETT  was  told  in 
simple  language  what  actions  to  take  to  correct  the  deficiencies.  The  NETT, 
however,  was  reluctant  to  make  changes  in  the  training  program;  therefore  the 
usefulness  to  the  NETT  of  the  changes  derived  from  the  modifications  job  aid 
was  not  fully  determined.  Some  of  the  recommendations  made  by  ARI  were  imple¬ 
mented  by  the  NETT  after  the  Director  of  DTD  issued  a  memorandum  reiterating 
certain  of  ARl's  recommendations  and  directing  the  NETT  to  incorporate  the 
recommendations  into  the  transition  training  program.  This  demonstrated  the 
potential  usefulness  of  the  modifications  job  aid  as  a  source  of  information 
for  specifying  alterations  to  training  programs  on  the  basis  of  observed  per¬ 
formance  deficiencies.  The  use  of  the  modifications  job  aid  greatly  simplified 
the  process  of  deriving  program  changes  from  program  deficiencies  and  thus  made 
the  analyst's  job  much  easier. 

In  response  to  recommendations  made  by  ARI  and  reinforced  by  DTD,  the 
NETT  added  demonstrations  to  some  lessons  where  there  had  previously  been  none 
and  required  instructors  to  adhere  more  closely  to  the  lesson  plans.  These 
changes  were  instituted  prior  to  administering  the  transition  training  for  the 
second  and  third  companies.  Evidence  that  these  changes  were  indeed  made  came 
from  the  comments  recorded  by  the  data  collectors  and  were  verified  by  informal 
contacts  with  the  data  collectors,  NETT,  and  OAFMS.  The  number  of  comments 
recorded  indicating  that  tasks  were  not  demonstrated  or  that  lesson  plans  were 
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not  followed  dropped  sharply  from  the  first  to  the  second  company  and  remained 
low  for  the  third  company.  The  number  of  comments  concerning  demonstrations 
was  81,  11,  and  13  for  the  first,  second,  and  third  companies,  respectively. 

The  number  of  comments  concerning  adherence  to  lesson  plans  was  27,  9,  and  8 
for  the  first,  second,  and  third  companies,  respectively.  The  sharp  decline 
in  the  number  of  comments  concerning  demonstrations  and  adherence  to  lesson 
plans  from  the  first  to  the  second  and  third  companies  parallels  known  changes 
in  the  training  process,  and  thus  demonstrates  the  sensitivity  of  the  TPE 
methodology  for  detecting  changes  in  the  training  process. 

Additional  indicators  of  the  sensitivity  of  the  TPE  methodology  to  program 
changes  were  not  available  because  some  of  the  changes  made  were  not  documented, 
making  it  difficult  for  the  analyst  to  verify  that  the  changes  had  actually 
been  made.  The  analyst  was  not  informed  when  changes  were  made  in  the  training 
program,  and  the  lesson  plans  were  not  revised  to  include  many  of  the  changes 
that  were  instituted.  This  reduced  the  ability  of  the  training  analyst  to 
assess  the  effects  of  the  various  changes  made  during  the  transition  training 
program. 

It  was  expected  that  as  changes  generated  by  ARI's  analysis  of  the  train¬ 
ing  program  were  implemented  by  the  NETT  that  the  percentage  of  tasks  for  which 
soldiers  received  first-time  GO's  would  increase.  Such  an  increase  would  sug¬ 
gest  that  the  changes  made  in  the  training  program  as  the  result  of  the  TPE 
analysis  were  increasing  training  effectiveness.  The  predicted  increase  in 
performance  over  the  three  companies  was  obtained.  The  proportion  of  tasks 
for  which  100  percent  of  the  soldiers  tested  received  first-time  GO’s  increased 
from  24  percent  for  the  first  company,  to  34  percent  and  53  percent  for  the 
second  and  third  companies,  respectively.  While  such  increases  may  be  due  in 
part  to  other  factors  (e.g. ,  reduction  of  standards  for  some  tasks  and  elimina¬ 
tion  of  some  task  requirements),  the  trend  toward  higher  first-time  GO  rates 
constitutes  indirect  evidence  that  the  changes  made  in  the  training  program 
from  one  company  to  the  next  increased  training  effectiveness  and  thus  supports 
the  usefulness  of  the  TPE  methodology  responsible  for  these  changes. 


SECTION  VII 


REVISING  THE  TPE  SYSTEM  ON  THE  BASIS  OF 
PROBLEMS  IDENTIFIED  DURING  THE  FIELD  TRIAL 


The  field  trial  of  the  TPE  system  during  the  Ml  OT-III  indicated  that  the 
system  could  be  used  by  Army  personnel  to  identify  training  program  deficien¬ 
cies  and  their  associated  causes  and  to  make  recommendations  for  changes  in  the 
training  program  that  when  implemented  reduce  the  number  of  performance  defi¬ 
ciencies  observed.  But  the  field  trial  also  identified  problems  in  using  the 
TPE  system  to  evaluate  large  scale  training  programs  such  as  the  Ml  OT-III 
transition  training  program.  In  the  paragraphs  that  follow,  solutions  to  some 
of  the  problems  encountered  in  using  the  TPE  system  during  the  Ml  OT-III  will 
be  proposed  and  changes  made  to  the  data  collection  forms  and  job  aids  will  be 
described  and  discussed. 

Motivation  of  the  Data  Collectors 


Many  of  the  problems  encountered  by  ARI  during  the  Ml  OT-III  arose  because 
some  of  the  essential  elements  of  the  training  evaluation  were  not  controlled 
by  the  training  analyst.  The  data  collectors  are  a  case  in  point.  The  data 
collection  team  was  trained  by  ARI  but  was  controlled  by  TCATA  during  data 
collection  activities.  The  training  analyst  visited  the  training  site 
periodically  and  talked  with  the  data  collectors  in  an  effort  to  determine  if 
the  observers  were  having  any  difficulties  in  completing  the  data  collection 
forms.  During  these  visits  to  the  field,  the  analyst  noticed  that  many  of  the 
data  collectors  seemed  to  lack  the  motivation  to  perform  the  data  collection 
task  well.  The  analyst  also  observed  some  of  the  factors  that  were  contributing 
to  this  lack  of  motivation,  such  as  too  many  forms,  poor  working  conditions,  and 
being  treated  as  unnecessary  by  the  NETT  and  the  unit.  However,  ARI  had  little 
control  over  these  factors  during  the  OT  and  thus  was  unable  to  change  these 
factors.  In  subsequent  applications  of  the  TPE  methodology,  the  following 
steps  should  be  taken  in  order  to  ensure  that  the  data  collectors  are  properly 
motivated.  In  the  TPE  workshops  the  importance  of  the  data  collection  task  to 
the  overall  evaluation  and  what  evaluation  can  accomplish  should  be  stressed. 
Data  collectors  should  be  forewarned  that  they  are  likely  to  be  treated  as  ad¬ 
versaries  by  the  instructors  who  deliver  the  training,  but  they  should  be  as¬ 
sured  that  their  job  is  as  important  as  that  of  the  instructors  and  that  they 
will  be  fully  supported  in  their  efforts.  If  possible,  the  data  collectors 
should  work  directly  for  the  analyst.  In  this  way  the  analyst  can  exert  con¬ 
trol  over  the  data  collection  activities,  ensuring  that  the  number  of  forms  to 
be  completed  by  any  one  data  collector  and  the  number  of  hours  spent  completing 
these  forms  are  limited  to  a  reasonable  level.  Likewise  the  analyst  can  ensure 
that  the  data  collectors  are  supplied  with  the  necessary  creature  comforts. 

Data  Flow  Problems  and  Suggested  Solutions 

One  of  the  problems  encountered  during  the  Ml  OT-III  was  that  information 
was  not  received  in  a  timely  manner  by  those  who  most  needed  it.  The  analyst 
sometimes  did  not  receive  the  evaluation  data  until  several  days  after  they 
were  collected,  and  the  NETT  did  not  receive  the  analyst's  recommendations  until 


several  weeks  after  the  recommendations  had  been  made.  In  order  to  ensure 
the  timely  flow  of  data  from  the  data  collector  to  the  analyst  and  from  the 
analyst  to  the  persons  responsible  for  instituting  changes  in  the  training 
program,  a  direct  line  of  communications  should  be  established  from  the 
analyst  in  both  directions.  In  order  to  speed  the  flow  of  data  to  the  analyst, 
the  originals  of  completed  data  forms  should  go  directly  from  the  data  collector 
to  the  training  analyst.  The  analyst  would  then  analyze  the  data  and  the 
results  of  this  analysis  would  be  made  available  to  other  organizations.  If 
other  persons  or  organizations  needed  the  raw  data,  they  would  have  to  obtain 
copies  through  the  TPE  analyst.  The  analyst's  recommendations  for  changing 
the  training  program  would  be  forwarded  directly  to  a  member  of  the  program 
staff  responsible  for  making  changes  in  the  training  program.  This  person 
would  check  into  the  possibility  of  making  each  of  the  changes  suggested  by 
the  analyst  and  would  inform  the  analyst  which  changes  were  made  and  which 
were  not. 

Other  than  speeding  up  the  flow  of  information  between  the  analyst  and 
the  person(s)  responsible  for  making  training  program  changes,  other  advantages 
may  accrue  from  a  direct  line  of  communications  between  the  analyst  and  the 
program  staff.  The  chances  for  the  analyst  to  convince  the  program  staff  to 
make  the  recommended  changes  are  increased  through  direct  contact,  as  are 
the  opportunities  for  determining  why  some  of  the  recommendations  were  not 
adopted.  Additionally,  personal  contact  on  a  regular  basis  between  the  analyst 
and  the  program  staff  may  reduce  the  animosity  that  tends  to  develop  between 
the  evaluator  and  those  whose  program  is  being  evaluated. 

Problems  in  Using  the  Data  Collection  Forms 

Perhaps  the  most  critical  problem  surfacing  during  the  Ml  OT-III  was 
the  tendency  of  many  of  the  data  collectors  to  treat  the  data  collection 
forms  as  simple  checklists,  forming  general  impressions  about  the  training, 
and  responding  to  the  TPE  items  on  the  basis  of  these  impressions  rather 
than  upon  the  objective  criteria  provided.  This  led  many  data  collectors  to 
mark  an  item  OK  and  record  no  comment  when  the  item  should  have  been  marked 
Not  OK  and  a  comment  recorded.  The  NCO  data  collectors  exhibited  a  general 
reluctance  to  record  their  observations  in  a  written  comment,  even  when  they 
judged  some  aspect  of  training  to  be  Not  OK.  In  an  attempt  to  correct  this 
problem,  the  TPE  data  collection  forms  and  the  observer's  job  aid  were 
modified  considerably. 


Revision  of  the  Data  Collection  Forms 

The  three  data  collection  forms  -  a  training  observation  form,  a  testing 
observation  form,  and  a  training  environment  data  collection  form  -  used 
during  the  Ml  OT-III  were  retained,  but  the  formats  were  changed,  new  items 
were  added,  and  many  of  the  old  items  were  revised.  The  training  observation 
and  training  environment  forms  used  during  the  OT  required  the  data  collectors 
to  make  OK-Not  OK  judgments  for  each  aspect  of  the  training  environment  or 
training  process  that  they  were  being  asked  to  observe.  Requiring  OK-Not  OK 
judgments  encouraged  data  collectors  to  rely  on  general  impressions  rather  than 


on  objective  data.  Therefore,  OK-Not  OK  judgments  were  eliminated  from  the 
revised  data  collection  forms.  All  of  the  items  on  the  revised  forms  are 
phrased  in  question  format  so  that  the  observer  can  address  each  item  with  a 
"Yes"  or  "No"  response.  The  wording  of  the  items  on  the  revised  forms  is 
quite  specific,  leaving  little  room  for  interpretation.  The  increased 
specificity  of  the  items  encourages  the  observers  to  attend  more  closely  to 
the  training  and  also  reduces  the  need  for  the  observers  to  clarify  their 
responses  with  comments.  The  need  for  comments  was  further  reduced  by  the 
addition  of  a  number  of  new  items.  Several  specific  new  items  replaced  one 
or  two  less  specific  old  items,  thereby  reducing  the  need  to  clarify  responses 
with  comments.  The  addition  of  the  new  items  increased  the  power  of  the  TPE 
system  to  obtain  specific  information  regarding  such  critical  training  events 
as  demonstrations,  practice,  and  feedback.  The  wording  of  some  items  was 
revised,  not  just  to  increase  specificity,  but  also  to  make  the  item  more 
easily  understood.  Items  using  such  specialized  terms  as  need-to-know, 
nice-to-know,  performance-based,  subject-matter  based,  level  of  reality, 
and  isolated  practice  were  revised.  The  specialized  terms  were  eliminated 
and  more  widely  understood  terms  were  substituted  in  their  place.  Two  items 
were  eliminated  because  they  were  ambiguous  and  produced  no  useful  data  during 
the  OT-III.  Overall  ratings  of  the  training  and  test  by  data  collectors 
during  the  Ml  OT-III  provided  no  useful  information  and  thus  were  not  included 
in  the  revised  data  collection  forms.  The  rating  scales  tended  not  to  be 
useful  because  of  an  extreme  leniency  bias  on  the  part  of  the  data  collectors 
and  a  general  reluctance  to  use  the  rating  scales  as  they  were  designed  to  be 
used.  A  question  concerning  the  approximate  number  of  soldiers  receiving  No-Go's 
provided  little  additional  information  during  the  OT  and  was  thus  eliminated 
when  the  forms  were  revised. 

A  fourth  data  collection  form  -  a  Training  Plan  Description/Training 
Events  form  -  had  been  designed  for  determining  if  training  is  conducted  as 
specified  in  the  training  plan.  The  form,  listing  the  major  training  events 
and  other  pertinent  information  abstracted  from  the  lesson  plans,  was  to  be 
used  by  data  collectors  during  the  Ml  OT-III  to  record  if  the  major  training 
events  were  conducted  as  planned.  However,  the  lesson  plans  developed  for 
Ml  transition  training  included  little  detail,  and  the  data  collectors  could 
see  no  clear  advantage  in  using  information  abstracted  from  the  lesson  plan 
over  using  the  lesson  plan  itself.  Because  the  form  was  not  perceived  as  being 
useful  by  the  data  collectors  and  because  the  analyst  felt  that  the  additional 
data  provided  by  using  the  form  would  be  outweighed  by  the  additional  data 
collection  burden  placed  upon  the  data  collectors,  a  decision  was  made  not  to 
use  the  form  during  the  Ml  OT-III. 

The  decision  not  to  use  the  Training  Plan  Description/Training  Events 
form  during  the  OT-III  was  based  on  factors  unique  to  the  OT  and  is  therefore 
not  indicative  of  the  general  usefulness  of  the  form  as  an  evaluation  instrument. 
To  encourage  subsequent  use,  the  form  was  revised  to  make  it  easier  to  use 
and  more  objective.  On  the  revised  form,  the  training  events  were  listed  more 
concisely,  making  it  easier  for  the  data  collector  to  record  which  of  the 
training  events  occurred  and  which  did  not.  As  with  the  other  TPE  forms,  the 
revised  form  did  not  call  for  OK-Not  OK  judgments,  but  merely  required  the 
data  collector  to  record  whether  or  not  a  particular  event  occurred. 


The  Revised  Job  Aids 


The  revised  data  collection  forms  and  the  items  comprising  them  are 
described  in  a  revised  version  of  the  observer's  job  aid  (Witmer,  Note  5). 

The  earlier  version  of  the  job  aid  (Witmer,  Note  3)  was  designed  specifically 
for  use  by  the  data  collectors  during  the  Ml  OT-III.  The  revised  job  aid  was 
written  so  that  it  could  be  used  by  anyone  who  was  given  the  responsibility 
to  observe  training  and  testing  for  the  purpose  of  collecting  evaluation  data. 
The  revised  job  aid  was  changed  considerably  from  the  earlier  version.  Besides 
the  changes  to  some  items,  the  addition  of  other  items,  and  the  deletion  of 
OK-Not  OK  judgments,  changes  were  made  in  the  basic  organization  of  the  job 
aid.  Whereas  the  earlier  job  aid  had  specified  the  format  of  each  worksheet 
and  the  items  appearing  on  that  worksheet,  the  revised  job  aid  provides  a 
suggested  format  and  lists  items  that  might  be  selected  for  inclusion  on  that 
worksheet.  This  allows  the  training  analyst  the  option  of  using  the  suggested 
format  or  developing  an  alternative  format.  It  also  offers  the  analyst  an 
opportunity  to  select  the  items  appearing  on  each  worksheet.  To  assist  the 
analyst  in  selecting  the  items  and  to  make  the  job  aid  easier  to  understand, 
items  relating  to  the  same  type  of  training  activity  were  grouped  together. 

For  example,  all  items  related  to  practice  were  grouped  under  the  "Practice" 
heading,  and  all  items  relating  to  feedback  were  listed  under  "Feedback."  To 
further  aid  the  analyst  in  selecting  items,  items  were  dichotomized  according 
to  level  of  difficulty.  An  asterisk  was  used  to  identify  items  that  require 
more  skillful  observation.  This  enables  the  analyst  to  tailor  the  worksheets 
to  the  skills  of  the  observers  who  will  be  using  them. 

The  modifications  job  aid  was  also  revised  (Kristiansen,  Note  6)  to  make 
it  compatible  with  the  revised  observer's  job  aid  and  easier  to  use.  For 
each  of  the  items  for  identifying  training  problems  listed  in  the  observer's 
job  aid,  the  modifications  job  aid  proposes  possible  solutions  to  the  problems. 
As  in  the  observer's  job  aid,  the  modifications  job  aid  lists  solutions  by 
worksheet  and  groups  solutions  relating  to  the  same  type  of  training  activity 
together.  The  table  of  contents  has  been  expanded  to  allow  the  analyst  to 
quickly  locate  the  suggested  solution  for  each  training  problem  identified 
on  the  observer's  worksheets.  The  revised  modifications  job  aid  has  been 
greatly  expanded  over  the  earlier  version  and  provides  much  more  detail  on 
how  to  modify  training  programs  on  the  basis  of  deficiencies  identified  during 
training. 

As  mentioned  earlier  a  set  of  criteria  that  could  be  used  in  evaluating 
lesson  plans  was  developed  and  used  in  evaluating  the  Ml  transition  training 
lesson  plans.  In  conjunction  with  the  development  of  these  evaluation  criteria 
a  first  cut  was  made  at  producing  a  job  aid  for  the  evaluation  of  lesson  plans. 
This  preliminary  job  aid  for  evaluating  lesson  plans  relied  heavily  on  the  use 
of  examples  from  the  Ml  transition  training  lesson  plans  to  illustrate  the 
types  of  problems  that  generally  occur  in  the  design  of  lesson  plans.  Following 
the  Ml  OT-III,  the  lesson  plan  evaluation  job  aid  was  rewritten  (Kristiansen 
and  Witmer,  Note  7)  omitting  the  examples  from  the  Ml  transition  training 
and  including  more  general  guidelines  for  the  evaluation  of  lesson  plans. 
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The  job  aids  described  above  provide  guidance  on  how  to  evaluate  lesson 
plans  (Kristiansen  and  Witmer,  Note  7),  how  to  observe  training  and  testing 
to  identify  training  program  deficiencies  (Witmer,  Note  5),  and  how  to  modify 
training  in  response  to  deficiencies  discovered  during  training  (Kristiansen, 

Note  6).  These  job  aids  alone  do  not  provide  sufficient  guidance  to  the 
training  analyst  for  conducting  a  complete  training  program  evaluation. 

For  example,  the  job  aids  do  not  provide  guidance  on  planning  the  program 
evaluation,  selecting,  training,  and  supervising  data  collectors,  interacting 
with  the  program  staff  and  other  participating  agencies,  or  procedures  for 
interpreting  and  reporting  the  data.  To  provide  guidance  to  the  training 
analyst  on  these  and  other  facets  of  training  program  evaluations  not  covered 
in  the  previously  described  job  aids,  an  analyst's  job  aid  was  developed 
(Kristiansen  and  Witmer,  Note  8). 

These  four  job  aids  taken  together  provide  methods  and  materials  that 
can  be  used  by  Army  personnel  in  evaluating  training  programs.  The  guidance 
provided  in  these  job  aids  is  much  more  detailed  than  that  provided  by  the 
Harless  Guidelines  which  the  job  aids  replace.  Providing  separate  job  aids 
for  lesson  plan  evaluation,  training  observation,  and  training  program 
modification  simplified  the  evaluation  task  by  making  it  easy  to  locate 
information  concerning  a  particular  evaluation  activity.  Further  simplification 
was  obtained  by  reducing  the  number  of  worksheets  to  be  completed  by  the  training 
observer  and  training  analyst.  The  Harless  Guidelines  included  eighteen  different 
worksheets  -  a  worksheet  for  almost  every  possible  training  evaluation  activity. 
The  sheer  number  of  worksheets  to  be  completed  made  the  evaluation  task  appear 
quite  formidable.  ARl’s  experience  during  the  Ml  OT-III  showed  that  many  of 
these  worksheets  were  not  necessary  for  conducting  a  training  program  evaluation. 
Further  examination  of  the  Harless  worksheets  following  the  Ml  OT-III  indicated 
that  much  of  the  information  called  for  by  the  various  worksheets  was  redundant 
and  that  the  kinds  and  amount  of  information  provided  by  some  worksheets  did  not 
justify  their  existence.  The  revised  job  aids  include  only  four  worksheets  or 
data  collection  forms. 

These  data  collection  forms  provide  the  necessary  guidance  to  the  training 
observer  to  ensure  that  the  training  data  needed  by  the  analyst  are  collected 
during  training  and  testing.  Although  the  training  analyst  may  wish  to  develop 
additional  forms  for  collecting  some  of  the  information  needed  to  evaluate  a 
particular  training  program,  no  other  forms  are  required  by  the  TPE  system. 


SECTION  VIII 


CONCLUDING  REMARKS 

The  TPE  system  for  evaluating  the  effectiveness  and  efficiency  of  training 
programs  as  described  in  the  revised  job  aids  offers  substantial  advantages 
over  other  methodologies  used  in  evaluating  training.  Like  evaluation  methods 
that  employ  specially  constructed  post-tests,  TPE  provides  an  objective  means 
of  determining  training  program  effectiveness.  But  unlike  post-tests,  TPE 
does  not  require  substantial  amounts  of  the  soldiers  time  over  and  above  what 
would  normally  be  spent  in  training.  And  while  post-tests  may  merely  indicate 
that  a  given  program  of  instruction  (POI)  is  effective  or  ineffective,  TPE  is 
capable  of  diagnosing  specific  problems  in  the  POI.  Like  questionnaire  methods 
of  evaluating  training,  TPE  seeks  to  identify  the  source  of  training  problems 
by  gathering  extensive  information  about  the  conduct  of  the  training  program. 

But  unlike  questionnaire  methods,  the  information  is  obtained  through  direct 
observation,  rather  than  from  second-hand,  post-facto  accounts  of  the  training 
program  by  the  program  participants. 

The  TPE  system  described  in  the  revised  job  aid  is  the  product  of  many 
months  of  developing  prototype  data  collection  instruments  and  job  aids  and 
testing  those  materials  in  the  field  with  typical  user  input  against  typical 
Army  training.  Through  field  testing  the  prototype  materials,  many  valuable 
lessons  were  learned,  which  were  incorporated  into  revisions  of  the  TPE  system. 
In  revising  the  TPE  system,  the  job  aids  were  refined  to  make  them  easier  to 
use  and  comprehensive  enough  to  provide  detailed  guidance  for  the  training  ob¬ 
server  and  training  analyst.  The  revised  job  aids  (Witmer,  Note  5;  Kristiansen, 
Note  6;  Kristiansen  and  Witmer,  Note  7;  Kristiansen  and  Witmer,  Note  8)  provide 
a  wealth  of  information  for  anyone  who  is  involved  in  the  evaluation  of  train¬ 
ing  programs,  and  are  particularly  useful  for  persons  responsible  for  evaluating 
training  programs  in  the  Army. 
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APPENDIX  A 


TRAINING  OBSERVATION  FORM 

Date _  Time  (from)  _  (to) 

Place  Nr  Trainees _ Nr  AIs 

Observer _  Subject  _ 

Instructor 


LESSON  PLAN 

Y  N 

Y  N 

Y  N 


Y  N 

Y  N 


Do  we  have  a  copy? 

If  so,  was  it  followed? 

Was  the  lesson  plan  changed  as  a  result  of  this  instruction? 
CONTENT 

Were  objectives  stated  in  performance  terms? 

What  were  they? 


HANDS-ON  TRAINING 

Did  lesson  involve  hands-on  practice? 

If  yes,  were  all  trainees  given  hands-on  experience? 

If  riot  all,  what  percentage? 

Did  hands-on  supervision  appear  adequate? 

Did  hands-on  time  per  trainee  appear  adequate? 

LECTURE/DEMONSTRATION 

Did  lesson  involve  lecturing?  Y  N  Demonstration? 

Did  lecture  appear  adequate?  If  no,  what  observations  did 

you  make  that  led  you  to  this  opinion?  (List  below) 

Did  demonstration  appear  adequate?  If  no,  what  observations 
did  you  make  that  led  you  to  this  opinion?  (List  below) 

Did  instructor  ask  for  feedback  from  trainees? 

Did  he  get  feedback? 

Did  he  handle  it  adequately? 

MISCELLANEOUS 

In  your  opinion,  was  the  total  time  adequate/excessive/ 
inadequate  for  this  lesson? 

In  your  opinion,  were  the  trainees  trained  to  criterion 
during  this  block  of  instruction? 

If  no,  what  observations  did  you  make  that  led  you  to  this 
opinion? 


Y  N 

Y  N 


Y  N 

Y  N 


Y  N 

Y  N 


Y  N 

Y  N 

Y  N 


Y  N 
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APPENDIX  B 

TRAINING  OPINION  QUESTIONNAIRE 


To  quality  control  future  XM1  training  program  development  efforts,  we 
need  your  opinion  regarding  the  training  you  have  just  completed.  Attached 

is  a  list  of  tasks  (numbered  1  thru  _ ) .  Please  respond  to  each  task  as 

follows. 


1.  Circle  YES  or  NO  opposite  each  task  to  indicate  whether  the  task  was  taught 
as  part  of  your  training. 

2.  If  the  task  was  taught  (YES),  answer  the  remaining  questions  for  that 
task  by  circling  the  number  that  best  reflects  your  opinion  of  the  training. 

1  =  Very  effective 

2  =  Effective 

3  =  Borderline 

4  =  Ineffective 

5  =  Very  ineffective 

3.  If  the  task  was  not  taught  (NO),  go  to  the  next  task  listed  and  repeat 
the  procedure. 

After  you  have  rated  each  task  according  to  the  above  procedure,  complete 
the  questionnaire  by  answering  the  last  two  questions.  Be  completely  frank 
in  making  your  responses  and  comments. 


POSITION 


US  Army  Research  Institute 
Fort  Knox  Field  Unit,  Fort  Knox,  Kentucky 


PT  5212a 


TRAINING  OPINION  QUESTIONNAIRE 


•ms  u 


Which  of  the  task(s)  rated  do  you  feel  require  additional  training  time? 
1. 


List  any  additional  task(s)  which  you  feel  need  to  be  included  in  the 
training  program. 


Comments : 


OBSERVATION  OF  TRAINING  EVENTS  -  PART  I  (Introduction/Overview/rrerequisites) 

XM1  Tng  39 


Class/ lossoh  Title 

Training  Locat ion/ Environment 


Class  Start  Time 


Class  End  Time 


19L  19K  A5E  63E 


Data  Collector  ID  II 


Instructor  ID  II 


COMMENTS 

(Continue  Below  if 
Necessary) 


Were  Students  Told 
the  Training  Objectives? 


Was  the  Purpose  of  the 
Training  Objective  provided 
to  the  students? 


Relationship  between  the 
Training  Objective  and 
other  events  was  explained? 


Positive  or  Negative 
Consequences  for  learning 
or  not  learning  were 
provided? 


Was  the  outline  of  class 
activities  and  schedule 
provided1 the  students? 


Questions  encouraged 
and  answered? 


ADDITIONAL  COMMENTS 


OBSERVATION  OF  TRAINING  EVENTS  -  PART  II  (Demonstration  and  Isolated  Practice) 

XMl  Ing  39 


Class/Lcsson  Title 


OBSERVATION 


Were  Tasks  Demonstrated? 

Was  Demonstration  conducted 
in  small  enough  steps? 

Was  the  isolated  practice 
at  a  high  reality  level? 

Isolated  practice  allowed 
for  a  range  of  examples? 

Students  were  provided 
feedback  on  their  actions? 

Faulty  performance  was 
identified  and  corrected? 

Was  sufficient  repetition 
allowed? 

Were  students  encouraged  to 
ask  questions? 

Student  questions  were 
answered? 

Were  Job  Aids  introduced  as 
Part  of  the  instruction? 

Level  of  reality  progressed 
from  Low  to  High? 


ADDITIONAL  COMMENTS 


FH  Form  2951-40 
(0T  29  Aug  80) 


Date: 


YES  NO  N/A  ASSESSMENT 
OK  I  NOT  OK 


COMMENTS 


Page  2  of  4 


OBSERVATION  OF  TRAINING  EVENTS  -  PART  III  (Ceneral) 

XM1  Tng  39 


Class/Losson  Tide 


OBSERVATION 


Instructor  followed  lesson 


AV  called  for  was  used? 


Training  materials  and 
liandouts  called  for  were  used? 


Instruction  was  logically 
sequenced? 


N/A  ASSESSMENT 
OK  I  NOT  OK 


Learning  Objectives  smoothly 
transitioned  from  one  to  the 
other? 


ations 


Need  to  Know  was  emphasized 
and  nice  to  know  minimized? 


Lessons  were  performance  based 
and  not  subject  matter  based? 


Students  actively  participated 


Instructor  provided  a  summary 


Sessions  start  and  end  on  time? 


Was  instructor  attitude 
positive? 


ADDITIONAL  COMMENTS 


XMl  Tng  39 

OBSERVATION'  OF  TRAINING  EVENTS  -  PART  III  (cont) 


OVERALL  RATING  OF  THE  CLASS  OR  LESSON 


Q  -  EXCELLENT 
Q  -  GOOD 
Q  -  FAIR 
Q  -  POOR 


No  performance  problems  predicted 
Few  performance  problems  predicted 
Considerable  performance  problems  predicted 
Widespread  performance  problems  predicted 


Summarize  FAIR  or  POOR  ratings 


OBSERVATION  OF  l-UAl.T ICE/TEST  EVENTS 

XMl  TNG  40 

Cl.i:;:. /Lesson  Ti  Lie 

19L  19K  4I>E  63E 

Page  1  of  J 

Clan: s/IiO .'.son  II 

Tra  i  n i  nij  l.oceLion/Knvironmcnt 

Date : 

Test  Start  Time 

Data  Collector  ID# 

Test  End  Time 

Instructor 

ID# 

Test  Basis?  _ 

jlndividual 

_ Crew 

Platoon 

Field  Performance 

Simulation 

Is  Test  Being  Scored? 

YES  NO  BY 

WHOM?  _ 

Is  Criteria  for  PASS/FAIL  Documented?  YES 

NO 

WHERE 

Have  the  Students  been 

given  the  criteria?  YES  __ 

NO 

Do  Guidelines  for  the 

Test  Administration  exist? 

YES 

NO 

WHERE  _ 

Is  this  remedial 

or  POI  Training? 

OBSERVATION 


Did  the  test  occur  soon  after  the 
completion  of  training? 


Were  Instructions  Clear? 


Are  PASS/FAIL.  Standards  Clear? 


Are  PASS/FAIL  Standards  Fair? 


Level  of  Reality  is  as  close  to 
real  world  as  possible? 


YES  l  NO  UNK 


COMMENTS 


Test  sequence  is  the  same  as  in 
real  world? 


Are  critical  discriminations  and 
responses  called  for? 


OBSEKVATICJN 


Kero  the  specifira  : 
applied? 


Arc  scorers  d i fccront  personnel 
than  instructors? 


Was  performance  contaminated? 


TRAIN I NO  ENVIRONMENT 

XM1-TNC-41 

Class/Lesson  Title 

19L  19K  45E  63E 

Class/ Lesson  if 

Training  Location 

Date: 

Number  of  Students 

Data  Collector  IDil 

Number  of  Instructors/AI' s 

Instructor  IDtf 

ENVIRONMENTAL  FACTORS 


Student-Instructor  Ratio 


Student-Equipment  Ratio 


Access  to  Instructor 


Access  to  Equipment 


OK  NOT  OK 


XM1-TNC-1 


DEMOGRAPHIC  PROFILE  DATA  SHEET / QUESTIONNAIRE 

..  .'1 1JXS:  fill  in  from  side  of  form  from  data  obtained  f-rom  personnel  records 
'  of  form  during  interview;  also,  during  interview  confirm  data  on  front  sid 


Icer.t  t.‘  .c-t  ion  number: 


Lasc/First/MI 


10.  MILITARY  EDUCATION 

Y  I.  (BASIC) 

Y  S  (AIT) 

Y  N  (MCS  SCH) 

Y  N  (XXI  SCH) 

11.  Pay  Gradej: 


iOtty  -j  Date: _ _ _ 

Day/Xonch/Year 


12.  Permanent  MOS 
(PMOS) : 


...  re  Departed  Unit  l  _ — 

Day/Month/Year 


CURRENT  I'M  IT 


(CO) 

(PIT) 


Rank: 

M 

• 

Other  MOS  (OMOS) : 

Classification  Battery  Scores: 

15. 

Basic  Active  Service  Date: 

6a  CO 

(Enter  Day,  Month,  Year, 
on.  appropriate  line) 

6f  CL 

15a  Before  I960 

-•b  Fa 

6g  GM 

15b  1950-1965 

cc  ST _ 

6h  SC 

15c  1956-1971 

oi  MM 

15d  1572-1978 

-  2  _ 

6j  Other 

15e  After  1978 

..eight : 

(Enter  exact  height  on 

16. 

.Birthdate: 

• 

appropriate  line) 

Day/Month/Year 

/  £*  odlow  3  0 
/ j  5'0"-5,A" 

17. 

Birthplace:'  (Enter  state 

On  aperopriate  line  or 

7c  5*5“ 
Id  o' 10 

-5’ 9" 

check  appropriate  Country) 

17a  •  North'  America 

?e  over 

o' 2" 

17b  South/Central 

America 

height ; 

(Enter  exact  weight  on 

17c  Carr ib bean 

appropriate  line) 

17d  Europe 

17e  Pacific 

on  SeL.-.- 

/  123  lbs 

17f  Other 

3b  1?)- 

.30  lbs 

He.  15i~ 

:75  lbs 

18. 

Physical  °rofi!e 

J[7C-20D  Vos 

VUat  l  ype 

Over 

2 oo  lbs  • 

*  k  .‘  !  i  r , 

Education: 

V.  -  /Months 

19. 

Total  Military 

Service : 

Years /Months 


«  •»  -- 


E 


XM1-TMC-1 


20.  Z„3  Tc  or:  _ 

21.  Security  Clearance: 


/ 

) 

Top  Secret 

21c 

( 

) 

Confidential 

21b 

t 

\ 

) 

Secret 

21d 

C 

) 

None 

••.hat 

is 

your  present  duty  position? 

( 

) 

Gunner 

22h 

( 

) 

63G  Mechanic 

22b 

( 

) 

Loader 

22i 

( 

) 

45k  Mechanic 

22c 

( 

) 

Driver 

22j 

( 

) 

34G  Mechanic 

22d 

( 

) 

Tank  Commander 

22k 

( 

) 

Ocher; 

22e 

/ 

V 

) 

63E  Mechanic 

Explain: 

22; 

/ 

V 

) 

45E  Mechanic 

221 

( 

) 

63C 

-22g 

< 

) 

63K  Mechanic 

22a 

( 

) 

4S1T 

What 

is 

your  Marital  Status? 

23a 

( 

) 

Never  Married 

23b 

( 

) 

Married  and  wife. is  with 

23c 

( 

) 

Married  but  wife  is  not  with 

23d 

( 

) 

Divorced  or  widowed 

23e 

( 

) 

Legally  Separated 

24.  How  much  longer  do  you  expecc.Co  remain  in  the  Amy? 

24a  (.  )  5  or  nore  years 

24b-  (  >  3—  4-years 

24c  (  >  2— 3- years 

24d  (  )-  2-  years 

24e  (  )  less  than  one  year 

25.  Kow  long  have  you  held  your  current  pay  grade? 

25a  C  )  -5  or  .tore  years 

25b.  (  ) — 3—  4 -years-- 

25c.  (  ) — 2— 3-years 

25d  (  )»..1„-.. 2-years 

25a  (  )  Less  than  nee- year 

26.  How  long  have  you  been'in  your  present  duty  position  (total  experience,  including 
ether  unit  assignments)? 

26a  (  )  Six  months  or  less 

26b  .  (  )  7  -  12  months 

26c  (  )  More  than  1  but  less  than  2 

26d  (  )  More  than  2  but  less  than  3 

26e  (  )  3  or  more  years 


?H  Fora  2951-1 
(OT  8  Apr  oO) 


XM1-TNG-1 


27.  How  long  have  you  been  a  member  of  your  present  crew,  group  or  squad? 

27a  (  )  Less  than  3  months 

27b  (  )  3  -  6  months 

27c  (  )  7-12  months 

27d  (  )  13-13  months 

27e  (  )  More  than  18  months 

23.  How  long  have  you  been  in  your  present  company,  troop  or  battery? 

23a  (  )  Less  than  3  months 

2Sb  (  )  3  -  6  months 

28c  (  )  7  -  12  months 

28d  (  )  13  -  18  months 

28e  (  )  More  than  18  months 

29.  How  long  have  you  been  in  this  battalion  or  squadron? 

29a  (  )  Less  than  3  months 

29b  (  )  3  -  6  months 

29c  (  )  7  -  12  months 

29d  (  >13-18  months 

29e  (  )  More  than  18  months 

30.  What  ancestry  do  you  consider  yourself? 

30a  (  )  Spanish  descent 

30b  (  )  American  Indian 

30c  (  )  Asian-American 

30d  (  )  Puerto  Rican 

30e  (  )  Philippino 

30f  (  )  Mexican-American 

30g  (  )  Eskimo 

30h  (  )  Aleut 

30i  (  )  Cuban-American 

30j  (  )  Chinese 

30k  (  )  Japanese 

301  (  )  Korean 

30m  (  )  Black 

30n  (  )  Other 

31.  Which  hand  do  you  use  for  doing  careful  work? 

31a  (  >  Left  hand 

31b  (  )  Right  hand 

31c  (  )  Either  hand 
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Whar.  kind  of  corrective  lenses  do  you  use  co  perform  in  your  duty  position? 

32a  (  )  Glasses  sometimes 

32b  (  )  Glasses  always 

32c  (  )  Contact  lenses  sometimes 

32d  (  )  Contact  lenses  always 

32e  (  )  Do  not  need  correction 


23-  How  many  dependents  (not  counting  yourself)  do  you  have? 

33a  (  )  None 

33b  (  )  One 

33c  (  )  Two 

33d  (  )  Three 

33e  (  )  Four  or  more 

34.  What  civilian  jobs  or  trades  have  you  worked  at  or  had  training? 

34a  (  )  Mechanic 

34b  (  )  Electrician 

34c  (  )  Construction 

.  34d  (  )  Machine  Operator 

34 e  (  )  Truck  driver 

35.  What  other  MOS  formal  training  do  you  have? 

35a .  Where  _ 

35b  What  Unit  _ 

35c  Duty  Position  _ 

35d  How  many  months  _ 

36.  What  primary  MOS  formal  training  do  you  have? 

36a  Where 

36b  Wnat  unit 

36c  Duty  Position  _ _ 

36d  How  many  months  _ 

37.  What  duty  MOS  formal  training  do  you  have? 

37a  Where  _ _  _ 

37b  What  unit  _ ~ _ 

37c  Duty  Position  _ 

37d  How  many  months  _ 

3S.  What  other  MOS  experience  do  you  have? 

38a  Where  _ _ 

38b  What  unit  _ 

38c  Duty  Position  _ 

38d  How  many  months  _ 
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What 

primary  MOS  experience 

do  you  have? 

39a 

Where 

39b 

What  unit 

39c 

Duty  Position 

39d 

How  many  months 

What 

duty  MOS  experience  do 

you  have? 

AOa 

Where 

AOb 

What  unit 

AOc 

Duty  position 

AOd 

How  many  months 

Al.  Are  you  an  advanced  individual  training  (AIT)  or  one  station  unit  training  (OSUT) 
gtaduate? 

Ala  (  )  AIT 

Alb  (  )  OSUT 

Ale  When: _ 

Inclusive  Dates 

Aid  Where:  _  _ 


A2.  What  ARTEP  experience  do  you  have? 

A2a  Where _ _ 

A2b  What  unit _ 

A2c  Duty  Position _ 

A2d  How  many  times _ 

A3.  What  Complete  Tank  Gunnery  experience  do  you  have? 

A3a  Where  _ 

A3b  What  unit  _ 

A3c  Duty  position  _ 

A  3d  How  many  times  ________________________ 

'»A.  What  maintenance  experience  do  you  have? 

AAa  Where  _ 

AAb  What  unit  _ 

AAc  Duty  position  _ 

AAd  How  many  months  _ 

A5.  ADDITIONAL  COMMENTS: 


DATA  COLLECTORS  NUMBER  _ 

DATA  COLLECTORS  NAME  _ _ _ 

LAST/FIRST/MI 

DATA  COLLECTORS  RANK  _ 
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TRAINING  AIDS  DATA  SHEET 

Class/Lesson  Title  _ 19L  19K  45E  63E 

Class/Lesson  #  _ _ 

Training  Location/Environment _  Date:  _ 

Test  Start  Time  _ _  Data  Collector  ID# 

Test  End  Time  _ _  Instructor  ID#  _ 

1.  List  training  aids  required  by  lesson  plan. 


2.  List  training  aids  used  during  class. 


3.  What  recommendations  do  you  have  for  more  effective  or  additional 
training  aids?  _ 


4.  How  affective  was  the  instructor's  use  of  the  training  aid(s)?  (Check 
one  and  explain) 

a.  (  )  Very  effective  _ 

b.  (  )  Somewhat  effective  _ 

c.  (  )  Somewhat  ineffective  _ 

d.  (  )  Very  ineffective  _ 
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14  OCT  80 
14  OCT  80 

INDIVIDUAL  TRAINING/PERSONNEL  STATUS  REPORT  XM1-TNG-38A 


A= GUNNER  B=LOADER  C=DRIVER  MECHANICS:  E=63E  F=45E  G=63H  H=63G  I=45K  J=34G  L=63C  M=43N  K=OTHER 

D=TANK  CMDR  MECHANICS:  E=63E  F=45E  G=63H  H=63G  I=45K  J=34G  L=63C  M=45N  K-OTIfflR 

NUMBER  OF  RECORDS=00065NK  CMDR 


XH1-TNG-38B 


TANK  CREW  PERSONNEL  STATUS  REPORT 

Class/ L.oson  Title  _ _ _ Dace  _ 

Ciass/Lesson  ii  _ _  Data  Collector  ID  If  _ 

Start  Time  _ _  Instructor  ID  If  _ ________ 

End  Time _  Crew  ID  0  _ 

Tank  USA  1/  _ 

Replacement  (Check  if  Yes) 
YES 


Tank  Commander  ID  1/ _ _  [  ] 

Gunnar  ID  If  _  [  ] 

Loader  ID  If  _ .  [  ] 

Driver  ID  If  _  [  1 


Reasons  for  regular  crew  members  not  being  present  for  training. 
ID# 

_  AWOL 

_ _  Hospital 

_  Emergency  Leave 


Confinement 
Other  (explain) 


STU3E;.T  OPINION  QUESTIONNAIRE 


DATE _ 19L  19K  45E  63E 

Please  tell  us  what  you  think  about  the  class  you  just  attended.  Be  Honest! 

3elow,  in  the  space  provided,  list  the  title(s)  of  the  class(es)  you  were  taught 
today.  In  the  columns  provided,  write  the  number  that  best  shows  what  you  think 
of  the  class. 

1  _  6000  (No  problems  -  everything,  was  clear) 

2  _  OKAY  (Few  problems  -  almost  everything  was  clear) 

3  _  BORDERLINE  (Barely  acceptable-an  average  class) 

4  _  BAD  (A  lot  of  problems-very  little  was  clear) 

5  _ TERRIBLE  (All  problems-nothing  was  clear) 


CLASS 

TITLE 


MANUALS 

AND 

PRACTICE 

TRAINING 

WORKBOOKS 

TIME 

METHOD 

CREW  STATION  TEMPERATURE  ASSESSMENT 
DATA  COLLECTION  FORM 


ORM  2951-44 
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1  MS  I  UUf.TOR  QULSTIOriNAIRE 

Ciass/Lesson  Title* _ 

Class/Losson  il _ 


19L  19K  4GE 


DATE: 


INSTRUCTOR  W 


Please  tell  us  what  you  think  about  the  class  you  just  taught.  Be  Honest! 

1.  Did  you  have  instructor's  material  for  each  task  you  taught? 


Did  not  for  these  tasks: 


2.  Do  you  understand  the  information  in  the  instructor's  materials? 


Didn't  understand  it  for  these  tasks: 


3.  Was  enough  information  given  to  you? 


Needed  more  info  for  these  tasks: 


4.  Was  the  information  in  the  instructor's  materials  technically  accurate? 


Inaccurate  info  for  these  tasks: 


9.  Did  you  have  standards  for  measuring  the  performance  of  students  in  the 
practice  exercises? 

Yes 

Didn't  have  standard  for  these  tasks: 
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TANK  CREW  FIRING  INTERVIEW 


EVENT  TITLE  _ 

EVENT  *  _ 

CREW  ID  #  _ 

COLLECTOR  ID  # 


INSTRUCTOR  ID  # 


1.  Did  you  experience  problems  with  putting  the  computer  into  operation? 
If  yes ,  explain: 


(  )  Yes 


(  )  No 


2.  Did  you  experience  any  problems  with  the  range  finder  during  firing? 
If  yes,  explain: 


(  )  Yes 


(  )  No 


4.  Were  there  any  problems  with  the  sights  (TIS,  GPS,  GAS)?  If  yes,  explain 
problem  and  crew  action: 


(  )  Yes 


(  )  No 


7.  Was  the  proper  ammo  indexed  for  each  firing? 


(  )  Yes 


(  )  No 


(  )  N/A 


8.  Did  the  tank  commander  fire  any  main  gun  rounds  from  his  override?  If 
yes,  why: 


imaged  ot  d 


14.  Was  the  driver's  hatch  closed  during  firing? 


(  )  Yes 


(  )  No 


15.  What  was  the  crew's  impression  of  the  computer? 


APPENDIX  E 


LESSON  PLAN  EVALUATION  CRITERIA 


1.  The  training  objective  should  specify  what  the  student  must  do  after  having 
been  trained. 

2.  Training  objectives  must  not  be  confused  with  job  requirements.  Training 
objectives  specify  behaviors  that  students  are  expected  to  exhibit  after  train¬ 
ing.  These  behaviors  are  not  necessarily  the  same  as  the  behaviors  that  are 
required  on  the  job. 

3.  A  training  objective  should  specify  the  conditions  under  which  the  student 
must  demonstrate  task  performance. 

4.  Conditions  stated  as  part  of  the  training  objective  must  be  clearly  train¬ 
ing  related  (rather  than  job  related). 

5.  The  training  objective  should  specify  the  standards  to  which  the  student 
must  perform.  The  standards  should  be  clearly  spelled  out  so  that  the  student, 
the  instructor,  and  the  training  evaluator  can  tell  the  difference  between 
performance  at  or  above  standard  from  performance  that  is  below  standard. 

6.  The  test  items  should  derive  directly  from  the  training  objectives. 

7.  Generally  the  test  should  require  the  student  to  perform  the  steps  speci¬ 
fied  in  the  training  objective. 

8.  The  conditions  specified  for  the  test  should  be  the  same  as  those  specified 
for  the  training  in  the  training  objectives. 

9.  ^he  standards  specified  for  the  test  should  be  the  same  as  those  specified 
in  the  training  objective. 

10.  The  instructions  for  administering  the  test  should  be  written  so  as  to 
ensure  standardization  of  test  administration  procedures  across  instructors. 

The  lesson  plan  should  outline  precisely  how  the  test  is  to  be  administered. 
Included  should  be  the  placement  of  personnel  and  equipment  and  test  adminis¬ 
tration  procedures  that  specify  the  testing  sequence  and  guide  the  evaluator 
in  testing  the  students  on  each  of  the  tasks  to  be  performed. 

11.  Instructions  to  the  students  concerning  how  the  test  will  be  conducted 
and  scored  should  be  included  in  the  lesson  plan.  These  instructions  should 
be  clear  and  complete  to  ensure  that  testers  do  not  have  to  add  instructions 
of  their  own. 

12.  The  lesson  plan  should  call  for  the  dissemination  of  enabling  knowledge 
and  specify  what  this  enabling  knowledge  consists  of.  (Terminology  or  other 
subject  matter  that  is  designed  to  enhance  students'  understanding  of  the  task 
to  be  learned  is  referred  to  as  enabling  knowledge.) 
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13.  The  lesson  plan  should  call  for  the  demonstration  of  the  task  (in  its 
entirety) . 

14.  When  there  are  subtasks  in  a  lesson,  demonstrations  should  be  required  for 
each  subtask,  and  an  integrated  demonstration  should  be  required  following 
subtask  training. 

15.  The  lesson  plans  should  describe  how  to  conduct  the  demonstrations. 

16.  Practice  activities  should  be  called  for  in  the  lesson  plan. 

17.  The  lesson  plan  should  specify  that  each  subtask  and  task  be  practiced. 

18.  The  lesson  plan  should  provide  guidance  to  instructors  that  tells  them 
what  to  look  for  during  practice  and  how  to  correct  faulty  performance  when  it 
occurs.  This  guidance  might  include  some  or  all  of  the  following:  (1)  speci¬ 
fication  of  aspects  of  the  tasks  that  might  be  expected  to  cause  problems  for 
students,  (2)  common  student  errors  on  the  task  being  practiced,  (3)  telling 
the  instructor  what  to  do  when  the  student  is  unable  to  even  start  doing  the 
task  (e.g.,  demonstrate  the  task  again),  (4)  telling  the  instructor  to  provide 
additional  assistance  when  student  progress  toward  the  objective  stops,  and 
(5)  directions  to  make  feedback  regarding  student  errors  immediate,  specific 
to  the  actions  performed,  and  free  from  harshness  or  ridicule. 

19.  The  lesson  plan  should  specify  that  practice  on  each  task/subtask  be  per¬ 
formed  by  each  student  to  a  specified  standard  (or  to  the  standard  specified 
in  the  training  objective). 
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